I have 2 308s and both have 1:10 twist rate barrels. Thinking about picking up another that's 1:12. Realistically what's the heaviest bullet I could expect to stabilize? It has a 26 inch barrel and I'll be running my handloads, likely pretty hot.

First, it's not the weight that determines bullet stability, but the length of the bullet. I have shot the 208 A-max out of an 18" and 20" 308 with 1:12" twist and they stabilize fine until transonic speeds (about 700-800 yards). But that 208 is not the best option for the longer range shooting in a 1:12" twist. They shot great inside of 600 yards. Knowing all that, I would keep the max bullet length about 1.400" or shorter. With the longer 26" barrel and higher velocity, it will give you some cushion.

I'd keep the stability higher than 1.4 on the Berger calculator below. You can see the various bullet lengths in the JBM link I posted to see what length the bullet is to calculate it.

https://bergerbullets.com/twist-rate-calculator/

http://www.jbmballistics.com/ballistics/lengths/lengths.shtml

I would hold out for a 1:11 or better. That’s just me. A 178 pushed hard in a 1:11 will perform at range. With a 1:12 I would think 168 and under to get optimal stability. I’m obviously generalizing because as chad said it’s length that matters. If you’re looking at a short bullet that performs great on game like a partition you’re not looking at long range anyway.

This would be mostly used for target shooting out to as far as I could make it work. Really it's just something new to tinker with. Other than working up a handload for my AR10, I'm out of projects. I'm sure you guys know what I mean.

Plus I've got an unused target scope sitting in the closet that's starting to feel lonely.

I really like 1-12 for all around since they shoot common hunting bullets so well and I've never had any problem stabilizing 180gr Partitions which are pretty long. We make 1-11.25 AR bbls and they seem to shoot most bullets from 125gr-180gr very well. I'm not at all a fan of 1-10 unless the rifle is a range toy I'm going to shoot 175gr Sierra MK in, that twist and bullet play very well together. There is a reason benchrest shooters opt for the slowest twist that will stabilize their chosen bullet, accuracy!

Look at the BC of the 155 gr Sierra Tipped Match King. For me, it squashed the 155 Lapua Scenar.

That much barrel, you can have it cooking coming out of the muzzle. I run it in my LR-308 at 2600 fps MV, and it'll run my whole range. You'll be much faster with a long barrel bolt action.

24" 1-10 175gr Sierra MK

My 1-12 twist shoots the 168 Hornady ELDM great. Never found a need to go heavier.