Does anyone know what is the best splitting text solution if I want to perform a search on the long RAW text?
I want to split the text into logical pieces \ paragraphs and search among them. I wonder if someone knows a lib \ hugging face model to do it.
I struggle to find a working solution.
Thanks!
I guess it depends what you mean by logical pieces and how your dataset looks like.
For example, a paragraph is “a short part of a text, consisting of at least one sentence and beginning on a new line. It usually deals with a single event, description, idea, etc.” So if that’s what you’re after, you could just split our string by \n.
But I vaguely assume you mean something else :).