what if you have a bunch of teams or apps that have different documentation patterns?
how much does this degrade over time, it beats leading models with that static data set but clearly this edge will degrade with data drift, how quickly does that happen?
also, any "this is 100x cheaper" blogpost means nothing if not discussing TCO (I know your team didn't write this.) I don't care what inference costs are if I don't know training/overhead costs. what's the breakeven point. and again, how long is this RAG stack going to be worth keeping, you beat 5.6 Luna but at some point un-tuned models will beat you, so this is a temporary solution that needs to be re-upped at some point. benchmarks against data drift would help there
- if you have teams that have totally different documentation patterns, yes i think a model trained one teams' docs probably won't directly transfer to another. but let's say you have a model trained on SEC 10-K filings, it should work well as new 10-K filings are added (since terminology & structure of these docs are similar) - on TCO, the training costs for the experiments you saw were <$200 - you bring up a good point on data drift, don't have concrete numbers/experiments here. something we should defn look into (though i imagine this also changes a lot based on use-case and company)