My main reason for playing with offline AI right now is to help me get further into the Computer Science curriculum on my own. (disabled/just curious)
I have seen a few AI chat characters with highly detailed prompts that attempt to keep the LLM boxed into a cosplay character. I would like to try to create fellow students in a learning curriculum. I haven’t seen anything like this yet, but maybe someone else here has seen this or has some helpful tips. I would like to prompt a character to not directly use programming knowledge from its base tokens and only use what is available in a Lora, or a large context, or a langchain database. I would like to have the experience of learning along side someone to talk out ideas when they have the same amount of information as myself. Like, I could grab all the information for a university lecture posted online and feed it to the AI, watch and read the information myself, and work through the quizzes or question anything I do not understand with the answers restricted to my own internal context region.
I have yet to meet a LLM that works decently locally. Wizard Uncensored is the closest, but the context length is too short, it keeps repeating itself after some time
Have you seen the great gatspy with Wizard too? That’s what always comes up when mine goes too far. I’m working on compiling llama.cpp from source today. I think that’s all I need to be able to use some of the other models like Llama2-70B derivatives.
The code for llama.cpp is only an 850 line python file (not exactly sure how python=CPP yet but YOLO I guess, I just started reading the code from a phone last night). This file is where all of the prompt magic happens. I think all of the easy checkpoint model stuff that works in Oobabooga uses python-llama-cpp from pip. That hasn’t had any github repo updates in 3 months, so it doesn’t work with a lot of newer and larger models. I’m not super proficient with Python. It is one of the things I had hoped to use AI to help me learn better, but I can read and usually modify someone else’s code to some extent. It looks like a lot of the functionality (likely) built into the more complex chat systems like Tavern AI are just mixing the chat, notebook, and instruct prompt techniques into one ‘context injection’ (-if that term makes any sense).
The most information I have seen someone work with independently offline was using langchain with a 300 page book. So I know at least that much is possible. I have also come across a few examples of people using langchain with up to 3 PDF files at the same time. There is also the MPT model with up to 32k context tokens but it looks like it needs server machine ram in the hundreds of GB to function.
I’m having trouble with distrobox/conda/nvidia on Fedora Workstation. I think I may start over with Nix soon, or I am going to need to look into proxmox, virtualization or go back to an immutable base to ensure I can fall back effectively. I simply can’t track down where some dependencies are getting stashed and I only have 6 distrobox containers so far. I’m only barely knowledgeable enough in Linux to manage something like this well enough for it to function. - suggestions welcome