The memory maximums are going to be more and more important when it comes to local AI applications.
Take language models for an example
To run a 30b model, you need 24gb of video ram to do it fully on the video card. That’s a nvidia 3090 or 4090 today. But in the grand scheme of things, 30b is small. They are going to get much bigger, especially when you want larger contexts which allow the AI to remember more about its interactions with you.
Apples memory is unified, so it can be system ram, or video ram. You’ll be able to easily load a 70b model into a MacBook with 64gb of ram for example, where you’d need 2 3090s or 4090s and a hefty PSU on a current Gen non Mac PC (if you even can with just that)
For the moment, things are better optimized for windows and nvidia hardware, but Apple is encroaching on this space, and their huge amounts of video memory will begin to unlock using and training larger and larger models with each hardware generation.
Expect to see nvidia starting to offer higher video ram cards as well for this exact reason. Maybe even cards tailored to that instead of gaming with really high amounts of ram.
The memory maximums are going to be more and more important when it comes to local AI applications.
Take language models for an example
To run a 30b model, you need 24gb of video ram to do it fully on the video card. That’s a nvidia 3090 or 4090 today. But in the grand scheme of things, 30b is small. They are going to get much bigger, especially when you want larger contexts which allow the AI to remember more about its interactions with you.
Apples memory is unified, so it can be system ram, or video ram. You’ll be able to easily load a 70b model into a MacBook with 64gb of ram for example, where you’d need 2 3090s or 4090s and a hefty PSU on a current Gen non Mac PC (if you even can with just that)
For the moment, things are better optimized for windows and nvidia hardware, but Apple is encroaching on this space, and their huge amounts of video memory will begin to unlock using and training larger and larger models with each hardware generation.
Expect to see nvidia starting to offer higher video ram cards as well for this exact reason. Maybe even cards tailored to that instead of gaming with really high amounts of ram.