In the 1.20260314.0 release, we shipped our first two Codex-authored Metal compute shaders, which improved LTX-2.3 video VAE decoding speed by about 2.4x on M1 through M4, and 4.7x on M5. On their own, the shaders delivered 4x and 8.3x speedups over the MPSGraph baseline, respectively.

3D convolution is one of those operations that had its day in the past. These days, much of its usefulness has been diminished by the rise of transformer architectures. But for video generation, 3D convolution is still a handy little tool for mixing information across time. There are other ways to do it, but it is handy, readily available, and well optimized on NVIDIA platforms.

It is also supported on Apple platforms, at least since 2022, through the MPSGraph API. In Draw Things, we use that API to support 3D convolutions for the video VAE decoders of video diffusion models such as Wan 2.x and LTX-2.x. We always knew we could probably do better ourselves, but it was never a priority.

Until LTX-2.3.

LTX-2.3 updated its video VAE to improve reconstruction fidelity. The new decoder is deeper, and therefore requires more FLOPs to complete. On M5, our old LTX-2.3 video decoding pipeline could take as long as the entire diffusion process combined.

In the experimental repo, Codex quickly gave us some initial numbers. The MPSGraph 3D convolution API could only reach about 1.1 TFLOPs, far below the roughly 12 TFLOPs we can achieve with our GEMM kernel on M5. Further analysis with Metal Frame Capture showed that the MPSGraph 3D convolution code path was not using the neural accelerator.

After that, the breakdown for how Codex should approach the problem was straightforward:

1. Give it access to the Metal Shading Language Specification PDF, and ask it to implement 3D convolution using the new 2D convolution tensor ops API.

2. Ask it to write unit tests to verify the results.

3. Give it development access to an M5 iPad so that it can benchmark the initial kernel.

4. Give it access to our old GEMM kernel so that it can understand the optimization target, namely reaching FLOPs parity.

There was some back-and-forth on how padding should be supported, but by the end of March 12, we had shipped a build with a Codex-authored, neural-accelerator-enabled 3D convolution shader to TestFlight testers.

I used that time to do a bit more research on the topic and found the MLX PR discussing the use of implicit GEMM to implement 3D convolution. The concept is similar to how we do that for 2D convolution. The next day, I gave Codex the PR link, our GEMM kernel implementation, and asked it to implement 3D convolution on an M3 Pro using implicit GEMM. Since we already had pretty good test coverage, Codex more or less auto-piloted to the finish line with a shader that quickly came within 10% of FLOPs parity. That was already 4x the performance of the MPSGraph implementation.

From there, I migrated the session to an M2 Ultra and asked Codex to benchmark it there and continue optimizing if needed. It initially showed some inefficiencies on M2 Ultra, which has more GPU cores, but Codex was able to ramp performance up quickly from about 10 TFLOPs to about 20 TFLOPs.

The build with the new optimized 3D convolution compute shader for pre-M5 devices went out to TestFlight on Friday. Today, we released it to the public.

While I still think we can do better with hand-rolled shaders, this kind of work usually takes us weeks, if not months. With Codex, it took 2 days.

It is here. Get used to it.

Qwen Image is a 20B-parameter image generation model that delivers next-level prompt adherence. Qwen Image Edit is a fine-tuned variant with editing capabilities. At its core, the model is a 60-layer MMDiT transformer combined with a fine-tuned Wan 2.x video VAE (from Alibaba’s video generation model).

As discussed in BF16 and Image Generation Models, MMDiT tends to gradually increase the activation scale during training. This was already an issue for Hunyuan and FLUX.1, which contain only 20 and 18 layers of MMDiT blocks respectively. For Qwen Image, activations can reach magnitudes on the order of ~20 million.

On Apple hardware from the M1/M2 era, it is generally better to keep most computation in FP16 to avoid BF16 emulation. However, due to Qwen’s drastic increase in activation range, while main activations can still accumulate in FP32, far more FP16 activations need scaling than in earlier models.

Activation Dynamics

In each MMDiT block, two pathways feed activations back into the FP32 main path:

• the out_proj result after attention,

• and the FFN result.

For earlier models, only the FFN computation required scaling. In Qwen, later layers must produce sufficiently large activations in both pathways to influence the FP32 path—causing FP16 overflow unless scaling is applied.

To address this, we adopted a more aggressive down-scaling strategy:

• The input to q/k/v projection is down-scaled by 8 (with RMS norm epsilon adjusted accordingly).

• The attention output is further down-scaled by 2, then up-scaled back after out_proj into the FP32 main path.

• For the FFN, we use a 32× down-scale factor for layers 0–58, and an even more aggressive 512× factor for layer 59.

With these strategies, we can run both Qwen Image and Qwen Image Edit in FP16 with minimal accuracy loss. A BF16 version is also provided, allowing only the critical layers requiring scaling to run in BF16 without scaling—minimizing the impact of BF16 emulation on older devices.

Read this post for a comparision of the activation scaling impact.

Releases @ Draw Things

Introducing Qwen Image Support

Qwen Image is the most powerful open-source image generation model to date, released by the Qwen team at Alibaba. We’ve been working hard to support it across the Apple ecosystem, and we’re happy to announce that it is now broadly available through the Draw Things app. From iPhone to Mac, Apple devices released within the past five years can run this st…

Read more

8 months ago · 10 likes · 2 comments · Authors of Draw Things

Video VAE

Qwen Image uses Wan 2.x’s video VAE to encode and decode latent space. Like FLUX.1’s VAE, it has a similar parameter count, but it employs causal 3D convolution for many operations. Naively applying the video VAE for first-frame decoding makes image generation slow: decoding a 1024×1024 image can take 5–6 seconds on an M3 Pro.

Looking deeper, however, Wan’s video VAE applies zero padding for previous frames during first-frame decoding. In these cases, full 3D convolution is unnecessary. By adjusting convolution weights/biases and switching to 2D convolution, we reduce decoding time to under a second for the same resolution.

Timestep-based Adaptive Layer-Norm

Of Qwen Image’s 20B parameters, about 7B are allocated to adaptive layer norm. In our SD3 optimization article, we described splitting the model to save VRAM. Unlike FLUX.1 or Hunyuan (which also use adaptive layer norm with MMDiT), Qwen Image’s adaptive layer norm depends only on the timestep.

This leads to an interesting implication: if we discretize timesteps between 0 and 1000, we only need to store 1001×718×3072 possible values, a lower number than the ~7B parameters required to generate them. In reality, timesteps in flow-matching models are not strictly discrete, but practitioners often fix the number of steps and shift values. By caching these projected conditions, we can avoid loading ~7B parameters into memory.

Note: This optimization isn’t necessary when the weights already reside in VRAM—the incremental FLOPs are minimal. It primarily helps when loading ~7B parameters into RAM is the bottleneck.

One area that has received little attention is the activation dynamics of diffusion transformers as they deepen. Architectures — particularly FLUX.1, a variant of MMDiT — tend to progressively increase activation scale. A common solution is to use BF16, which has a much larger dynamic range. This is one reason BF16 has gained popularity in image generation models.

However, BF16 brings its own challenges. Its reduced mantissa (compared to FP16) can lead to accuracy issues, and support for BF16 on older Apple Silicon (M1/M2) is limited. Software emulation only matured in macOS 15 — and even then, it’s roughly 50% slower than FP16 on these platforms.

Over the past year, Draw Things has refined its FP16 support to enable efficient execution of large diffusion transformer models on M1/M2, often achieving performance comparable to M3/M4. In this post, we’re sharing our general approach and model-specific tuning strategies to make FP16 viable. Our hope is to help extend support for cutting-edge models to a wider range of edge devices — especially for users without hardware-accelerated BF16 or who find BF16 accuracy unsatisfactory.

FP32

In diffusion transformers, a key challenge lies in the final layer normalization prior to projecting back into the latent space. This layernorm allows the preceding activations to scale freely, often beyond FP16’s representable range.

To address this, we upcast the main activation accumulation path to FP32. This sidesteps dynamic range limitations without significant performance cost — since element-wise operations (including layernorm) are not the main bottleneck in image or video generation.

FP16 & Transformer Block

For all MMDiT / DiT variants we’ve encountered, the main activation is routed through an adaptive layernorm before entering the transformer block. This presents a clean boundary where we can safely convert activations to FP16 and run the rest of the block in FP16.

For many models — such as the Wan 2.1 series — this is sufficient. But in some MMDiT variants with large MLP intermediate dimensions, additional care is needed.

Scaling in MLP Layers

The MLP layers often include a wide projection before collapsing back to the hidden dimension. The down-projection can lead to activation overflow in FP16. To mitigate this, we apply conservative scaling factors (typically ⅛ or ¼).

These factors are conservative by design: FP16’s 10-bit mantissa offers ~3 bits more precision than BF16, so a scaled FP16 activation (e.g., by ⅛) often retains more numerical fidelity than an unscaled BF16 value.

In these paths, we upcast to FP32 after the final GEMM in the MLP and perform the adaptive layernorm gating in float32.

Attention Scaling Strategy

Attention operations typically apply a scaling factor of 1/√k. In our FlashAttention implementation, accumulation occurs in FP16, which may still result in range issues.

We’ve found that applying the scaling factor before the attention (rather than fusing it inside the kernel) helps mitigate overflow and preserves numerical stability for our particular implementation.

Exact Configurations

Below are the exact FP16 tuning configurations we use in Draw Things for various models. These adjustments allow for stable FP16 inference without requiring full BF16 support:

FLUX.1

• Activation scaling factor: 8

• Scaled layers: Double stream blocks 17, 18 (0-indexed)

• Implementation Details

Hunyuan

• Activation scaling factor: 8

• Scaled layers: All double stream blocks

• Implementation Details

Wan 2.1 14B

• Activation scaling: Not needed

• Pre-scaling: Applied for attention

• Implementation Details

HiDream

• Activation scaling factor: 4

• Scaled layers: Double stream blocks 13, 14, 15 (0-indexed)

• Pre-scaling: Applied for attention

• Implementation Details

Metal FlashAttention underpins Draw Things’ claim of fastest image generation inside the Apple ecosystem. It conserves system memory, it is fast and it supports a wide-array of devices with the oldest being iPhone 12 from more than 4 years ago.

Back in September, Philip Turner and I released Draw Things with Metal FlashAttention 2.0. Since then, we’ve integrated not only the forward pass (useful for inference) but also the experimental backward pass (useful for training). Combining together, Draw Things is the only efficient application on macOS / iOS that supports both inferencing and fine-tuning FLUX.1 [dev], a 11B-parameter, state-of-the-art image generation model. This major version upgrade delivers:

• Up to 20% faster inference on newer hardware such as M3 / M4 / A17 Pro;

• Carefully tuned memory precision / register precision choices to make FP16 inference more accurate and less prone to NaN errors;

• Backward pass implementation that is up to 19% faster than naive implementations to support efficient training on Apple devices;

• Better tuned parameters to deliver efficient inference and training for larger head dimensions;

• Switch to runtime code generation for better compiler compatibility and ease-of-integration;

• Support of BFloat16 emulation, with a slight deviation from certain rounding rules to run more efficiently on older devices;

• Keeping performance consistent with a wide-array of sequence lengths and head dimensions (minimal performance cliffs).

Translating these gains into real-world numbers, we see up to 20% improvement on inference for FLUX.1 models on M3 / M4 devices, up to 20% improvement on inference for SD3 / AuraFlow models on M3 / M4 devices. Similar improvements for SD3 / AuraFlow for older hardware and around 2% improvement on older hardware for FLUX.1 models.

Compared to other implementations, FLUX.1 integrated inside Draw Things is up to 25% faster than mflux implementation on M2 Ultra for each iteration, and more for end-to-end times; it is up to 94% faster than ggml implementations (also known as gguf format). SD Large 3.5 integrated inside Draw Things is up to 163% faster than DiffusionKit implementation for each iteration (on M2 Ultra).

mflux: 0.5.1, DiffusionKit: 0.5.2, mlx: 0.21.1, ComfyUI: v0.3.8+PyTorch v2.6.0.dev20241218

On the training side, training SDXL LoRA at 1024x1024 now is 2% faster than our previous implementation in Balanced mode. There is no comparison for training FLUX.1 LoRAs on macOS, our implementation scores 9s per step per image at 1024x1024 resolution on M2 Ultra.

With the release of Metal FlashAttention 2.0, we invite the community to collaborate and extend this implementation to more downstream frameworks. Our reference Swift implementation is available at: https://github.com/philipturner/metal-flash-attention. Our C++ implementation is available as part of ccv: https://github.com/liuliu/ccv/tree/unstable/lib/nnc/mfa.

Appendix

Comparison with other SDPA (Scaled-dot Product Attention) kernel implementations (MLX, Apple MPSGraph). See raw data at https://docs.google.com/spreadsheets/d/1NHzYHcqtH5xb18trn9NyTc1EeSfXZen9C7E7vsPb_lI/edit?usp=sharing

End-to-end benchmark data is available at https://docs.google.com/spreadsheets/d/1A8xC2_wh_Nwc5p2uvNMnKMtN4kkJac1E764XHrADpBs/edit?usp=sharing

Model Conversion

Over the past year, we’ve significantly streamlined our model conversion workflow. What used to take weeks with Stable Diffusion 1.4 now takes about a day. For example, we implemented our FP16 version of SD3 Medium on June 13th, 24 hours after the release.

To deploy cutting-edge image/text generative models to local devices, we use Swift implementations that compile natively on these platforms. This involves translating Python code, typically written in PyTorch, into Swift. We begin this by setting up the correct Python environment, creating minimal viable inference code to correctly call the model, inspecting the result, and then implementing the Swift code.

PythonKit has been essential for our conversion work, allowing us to run Python reference code directly alongside our Swift reimplementation. The first-class support of s4nnc on CUDA systems also enables us to run our Swift reimplementation on Linux systems with CUDA, which is often the most hassle-free environment for running PyTorch inference code.

Our reimplementation generally involves rewriting the PyTorch model into a more declarative Swift model and comparing outputs layer by layer. This is particularly straightforward with transformer models, where each layer follows the same architecture.

Our implementation: https://github.com/liuliu/swift-diffusion/blob/main/examples/sd3/main.swift#L502-L661

SD3 Ref: https://github.com/Stability-AI/sd3-ref/blob/master/mmdit.py#L11-L619

Model Quantization

Deploying large models to local devices often requires weight quantization. For image generative models, we carefully balance quality and size trade-offs. With Draw Things, we ensure all our quantized models are practically “lossless.” We focus on sensible reductions that maintain compatibility across a wide range of devices rather than pushing for the smallest possible model size.

Currently, s4nnc supports limited quantization options, including 4-bit, 6-bit, and 8-bit block palletization as our main schemes. For diffusion models, we use the mean squared error metrics of the final image between quantized and non-quantized models to guide our decisions. We selected 8-bit quantization for SD3 Medium and 6-bit for the T5 encoder.

Model Optimization

Unlike the UNet in SDXL/SD v1.5, SD3 Medium uses straightforward transformer blocks, limiting optimization opportunities — especially regarding FLOPs. However, we managed to split the model to reduce peak RAM usage during the diffusion sampling process to approximately 2.2 GiB for the quantized model (around 3.3 GiB for the non-quantized model).

This is possible by observing that while adaptive layer norm blocks are minimal in FLOPs, they have a high parameter count, around 670M. Since the input for the adaptive layer norm includes timestep conditioning, we cannot reduce FLOP computation. However, since there are no dependencies on model intermediate activations, we can batch the adaptive layer norm computation of every timestep to the beginning of diffusion sampling all at once, converting matrix-vector multiplication to matrix-matrix multiplication, which is slightly more efficient.

Thanks to these optimizations, we implemented the fastest SD3 Medium model inference on macOS, iOS, and iPadOS systems with minimal RAM usage and successfully shipped it to real users within a practical app.

Future Directions

Our implementation can provide valuable feedback into the training process. Moving forward, we aim to conduct more research and ablation studies to explore:

1. Optimal parameter count distribution for adaptive layer norm — could we allocate fewer parameters here, and more to MLP/QKV projection?

2. Comparing more quantization schemes to identify per-layer improvements and establishing an unbiased prompt dataset for the future data-free fine-tuning.

3. Leveraging torch.compile to rewrite the PyTorch model in Swift, all from within Swift using PythonKit.

We are excited to continue our research and share our development work in the future.

However, since these advancements were primarily within the CUDA ecosystem, model fine-tuning was largely exclusive to powerful NVIDIA servers and PCs.

With the release of Draw Things version 1.20231004.1, we’ve extended the capability to fine-tune large image generation models like Stable Diffusion v1, v2, and XL to iPhone, iPad, and Mac.

Benefits of On-device Fine-tuning

On-device large model fine-tuning offers even more possibilities for AI-assisted creative workflows. Whether you’re using 3 to 4 photos to introduce a new identity to the model or hundreds of your artworks to teach it a new style, on-device fine-tuning ensures privacy and offers limitless customization. Fine-tuning on personal hardware provides a broader range of choices, from base model selection and image captioning to denoising schedules and learning rates. These training recipes are essential components of the creative process, not just for experimentations.

The Path to On-device Fine-tuning

Draw Things adopted a specific fine-tuning strategy known as LoRA. Specifically, our LoRA approach overlays the LoRA network on both linear and convolution layers (known as LoCon in Stable Diffusion community).

Our LoRA method also builds upon our Metal FlashAttention and JIT weight dequantization work, enabling us to train LoRA on quantized model weights directly, known as QLoRA.

For the first time ever, this setup allows the SD v1 model to be fine-tuned (at a 512x512 resolution) on an iPhone 15 Pro, with peak memory consumption around ~6GiB, including model weights. For the 3.5B parameter SDXL, our approach uses peak memory of approximately ~10.3GiB, making it possible to fine-tune such a large model on an iPad.

While the main network operates at FP16, the LoRA network runs at FP32 during training. This distinction stabilizes the training process, even with higher learning rates (up to 1e-3). Test users have reported that as few as 500 steps at learning rates of 1e-4 or 1e-3 are sufficient to introduce a new concept to the model. A lower learning rate combined with extended training steps allows the model to absorb more details from training samples.

Our method is also efficient. For instance, 500 steps at a 512x512 resolution with SD v1 on an iPhone takes about an hour, while on an iPad M2 or Mac Mini, it’s just ~20 minutes. Fine-tuning the SDXL at the same resolution and step count on an M2 Ultra takes 14 minutes. This speed makes it not only feasible but also practical to fine-tune models on personal devices for both professional and recreational purposes.

Over the past two weeks, we’ve collaborated closely with our community to test the LoRA training feature. The results have been impressive, with consistent character creation and “helper” LoRAs for hand and finger fixes. We’re eager to see what our users will come up with next.

The Future

At the beginning of our “Stable Diffusion on an iPhone” project, we estimated that about 50% of performance potential remained untapped for the inference code. This gap has since been closed with our Metal FlashAttention work.

Our training code is far from optimal. We believe there’s at least a 50% speed increase attainable, and we can reduce RAM usage by another 30% without additional quantization efforts and minimal impact on speed.

I’m also excited about the prospect of more creative control over the fine-tuning process, including features like ControlNet signal injection, combined base model and existing LoRA, and co-training with textual inversion.

Our implementation on iPhone supports the particular configuration at network dim = 8, 8-bit base SD model, and no text model co-training.

The fine-tuning uses AdamW optimizer with betas at 0.9, 0.999. Epsilon at 1e-8 while decay at 0.001.

With this release, we’ve also enabled LoRA export, allowing users to share their trained LoRAs on model-sharing sites.

Until now, most algorithmic innovations and improvements have occurred on NVIDIA CUDA hardware. It makes sense when most AI-related computing happens on server-side. However, as we move the compute closer to the edge, how to drive the same algorithmic innovations and improvements to one of the most used hardware platforms captured our imagination.

Over the past few months, Philip Turner and I worked closely to integrate his Metal FlashAttention into the Draw Things app. With version 1.20230807.0 of the app, it generally cuts the image generation time by half, often bringing superior experience than the cloud, with added benefits of privacy and freedom.

Metal FlashAttention

Metal FlashAttention comprises Metal compute shaders optimized for operations commonly found in large image generation and language models. That includes thin matrix multiplications (e.g. [4096, 320] x [320, 320]), scaled dot product attention (the heart of multi-head attention or transformers) and layer normalization. It stands as an open-source alternative to Metal Performance Shaders (MPS).

GEMM

GEMM computations, typically found in the Stable Diffusion variant of models (v1, v2, XL), don’t hit the sweet spot of Apple’s Metal Performance Shaders or MPSGraph implementation. Metal FlashAttention leverages the simdgroup_async_copy API (since A14), an undocumented hardware feature that overlaps compute and load instructions.

FlashAttention

Inspired by the FlashAttention project, Metal FlashAttention aimed to improve both latency and memory footprint. At the core, it is pretty easy to understand why: scaled dot product attention is concisely demonstrated by the two lines of code from the PyTorch documentation:

attn_weight = torch.softmax(scale * (Q @ K.transpose(-2, -1)))
return attn_weight @ V

There’s no need to to materialize the full Q @ K.transpose(-2, -1) matrix before computing the final result. Naively, you simply need one-row per Q @ K.transpose(-2, -1) to compute softmax and then do the final matrix multiplication. This approach, often referred to as attention slicing, has been attributed to the further performance improvements in apple/ml-stable-diffusion (named as SPLIT_EINSUM_V2).

The original FlashAttention in CUDA (by Dao AI Lab) focused on both forward and backward pass. Metal FlashAttention paid particular attention to the forward pass (inference). We made several optimizations to the original FlashAttention on the inference path, some of these optimizations are concurrently adopted in the FlashAttention v2 release. These optimizations decreased the total number of computations and increased numerical stability. We also made a unique block-sparse algorithm that automatically detects sparsity in the attention matrix. This approach allowed a single shader to handle sparse, causal, or irregular masks, especially masks that change dynamically at runtime.

The resulting speedup is not just measurable in percentages — it represents orders of magnitude. In the bottom graph, MPS performance was excluded as it couldn’t complete the benchmark in a reasonable time frame. It maxed out at 2000 GFLOPS (top), while MFA soared an order of magnitude higher (bottom).

Real-world Impact

The GEMM kernel of Metal FlashAttention has been integrated into the 1.20230726.0 release of the Draw Things app. The community has confirmed our claim of 10-30% performance improvements over many devices.

The full Metal FlashAttention integration, including GEMM with fused bias, scaled dot product attention with fused multi-head output projection, and custom layer normalization has gone through extensive testing & benchmarking.

Comparing w/out Metal FlashAttention, image generation latencies are roughly halved (43–120% faster). This speedup is consistent across a wide range of Stable Diffusion architectures, several device families (iPhone 12 and above, M1 and above) and every image resolution.

Comparing to CoreML implemented Stable Diffusion v1.x, v2.x with Apple Neural Engine, GPU-accelerated Metal FlashAttention slightly trailing behind ANE performance on M1 / M2 base model (~12.8s with ANE and ~15.2s with MFA, at 25 steps, 512x512) while outperforms them on M1 Pro / M2 Pro and above. With the v2.0 model on A16 chip (Stable Diffusion v1.x with CoreML cannot be run on iPhones without quantization), MFA trailing behind ANE performance (~26.3s with ANE and ~34s with MFA at 25 steps, 512x512) by ~22%.

Comparing to CoreML implemented Stable Diffusion v1.x, v2.x and XL with GPU, Metal FlashAttention outperforms in wider margin on M1 Pro / M2 Pro and above models (usually around 20% to 40% faster than CoreML GPU, ORIGINAL configuration). CoreML GPU implementation outperforms ANE implementation on these devices.

Metal FlashAttention is the kind of optimization that raises the tide for us all. Unlike our CoreML integration, Metal FlashAttention improves performance across all image generation resolutions and with any given prompt length, Whether dealing with the standard 77-token prompt or thousands of tokens prompt, every workflow benefits from Metal FlashAttention’s integration. By integrating into our low-level framework directly, there is no first-time-model-loading cost, nor first-time-model-conversion cost associated with CoreML or other alternative runtimes.

Above comparisons were done with Swift CoreML Diffusers app from macOS App Store (v1.1), with the exception of A16 performance (done with the CoreML implementation inside Draw Things app). SDXL comparison was done with Diffusers app built from source (commit: 4eab4767) with Xcode 15 beta 5 on macOS Sonoma beta 5, Release configuration, with FP16 SDXL Base CoreML model. Diffusers on iPad was built from source (commit: 4eab4767) with Xcode 14.3.1, Release configuration. Preview and Safety Checker were disabled. Both are measured with minimum of the 2nd and 3rd run, with Preload On in Draw Things (with exception of A16 / iPhone 14 Pro (too little RAM to preload the models). iPad Pro M2 is a 8GiB configuration so the Preload is automatically off for SDXL). All measurements are done with 25 steps.

Above comparisons were done with following hardware specs: Mac M2 refers to Mac Mini M2 with 16GiB RAM and 10 GPU cores. M1 Pro refers to MacBook Pro M1 Pro with 16GiB RAM and 14 GPU cores. M1 Max refers to Mac Studio M1 Max with 32GiB RAM and 24 GPU cores. M2 Ultra refers to Mac Studio M2 Ultra with 192GiB RAM and 76 GPU cores.

Sprinting Forward

With Metal FlashAttention integrated into the Draw Things app, and our community enjoys faster image generation time, we’re eager to see Metal FlashAttention get integrated into many other applications and frameworks to empower local inferences from image generation models to large language models in the Apple ecosystem.

The Metal FlashAttention project, authored by Philip Turner and sponsored by Draw Things, is open-source and can be found at https://github.com/philipturner/metal-flash-attention under MIT license.