1 prompt to rule them all - NOT
I have been using my Git Commit generator Using AI to build AI for well over a month now and love it. Reducing the tax of software development is the holy grail right?
Mistral released a new model Mistral: Devstral Small which OpenRouter where offering for free. Quick swap out and $$ savings for me!!.
But alas no!
The system prompt in Iamcommitted is pretty basic, but I had tuned it with ChatGPT to give me a decent response. But switching to Mistrel, suddenly the responses are multi line, verbose and unusable.
I dont attribute this to a problem with Mistrel, but the reality that #promptengineering really is a thing. But how do I fit #promptengineering into my standard developer workflow? I want #TDD
I want to iterate in my IDE. I want #promptengineering with #tdd, I dont want trial and error in a chat interface. 🤔
Insights
- Openrouter was a straight swap out for OpenAI API which was fantastic
- Change to MistralDev has made the commit message verbose!!! Need new system prompt for running IamCommitted with MistralDev but it’s free!!!
- poses question
- How do I test a set of system and user prompts against different models?
- Is this more of a data scientist workflow
- Are there decent frameworks for test and learn that fit an engineer’s workflow?
- How do I test a set of system and user prompts against different models?
written by a human