• bumblebeer@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    10 days ago

    Well yes, I’m not saying it’s a free lunch. The point is a large up-front investment buys you unlimited tokens in perpetuity for no more than the cost of electricity to power the cluster — which can be surprisingly cheap for UMA machines.

    I’ve swapped off API models entirely and have been able to do a lot more useful work since I don’t have to worry about token cost or usage limits. Granted it’s slower per task (less tk/s), but overall much more productive, and private.