pinned
Build error
13
Pdfitdown
๐
Convert (almost) everything to PDF!
Clelia Astra Bertelli
as-cle-bert
AI & ML interests
Biology + Artificial Intelligence = โค๏ธ | AI for sustainable development, sustainable development for AI | Researching on Machine Learning Enhancement | I love automation for everyday things | Blogger | Open Source
Recent Activity
liked
a model
21 days ago
facebook/dinov2-small
posted
an
update
6 months ago
Let's pipe some ๐ฑ๐ฎ๐๐ฎ ๐ณ๐ฟ๐ผ๐บ ๐๐ต๐ฒ ๐๐ฒ๐ฏ into our vector database, shall we?๐ค
With ๐ข๐ง๐ ๐๐ฌ๐ญ-๐๐ง๐ฒ๐ญ๐ก๐ข๐ง๐ ๐ฏ๐.๐.๐ (https://github.com/AstraBert/ingest-anything) you can now scrape content simply starting from URLs, extract the text from it, chunk it and put it into your favorite LlamaIndex-compatible database!๐ธ๏ธ
You can do it thanks to ๐ฐ๐ฟ๐ฎ๐๐น๐ฒ๐ฒ by Apify, an open-source crawling library for python and javascript that handles all the data flow from the web: ingest-anything then combines it with ๐๐ฒ๐ฎ๐๐๐ถ๐ณ๐๐น๐ฆ๐ผ๐๐ฝ, ๐ฃ๐ฑ๐ณ๐๐๐๐ผ๐๐ป and ๐ฃ๐๐ ๐๐ฃ๐ฑ๐ณ to scrape HTML files, convert them to PDF and extract the text - hassle-free!๐ธ
Check the attached code snippet if you're curious of knowing how to get started๐ฌ
PS: Don't tell anybody, but this release also has another gem... It supports OpenAI models for agentic chunking, following the new releases of Chonkie๐ฆโจ
If you don't want to miss out on the new features, leave us a little star on GitHub โก๏ธ https://github.com/AstraBert/ingest-anything
And join our discord community! โก๏ธ https://discord.gg/kDqHNjks
posted
an
update
6 months ago
Hey there, ๐ถ๐ป๐ด๐ฒ๐๐-๐ฎ๐ป๐๐๐ต๐ถ๐ป๐ด ๐๐ญ.๐ฌ.๐ฌ just dropped with major changes:
โ
Embeddings: now works with Sentence Transformers, Jina AI, Cohere, OpenAI, and Model2Vec
All powered via ๐๐ต๐ผ๐ป๐ธ๐ถ๐ฒโ๐ ๐๐๐๐ผ๐๐บ๐ฏ๐ฒ๐ฑ๐ฑ๐ถ๐ป๐ด๐.
No more local-only limitations ๐
โ
Vector DBs: now supports ๐ฎ๐น๐น ๐๐น๐ฎ๐บ๐ฎ๐๐ป๐ฑ๐ฒ๐
-๐ฐ๐ผ๐บ๐ฝ๐ฎ๐๐ถ๐ฏ๐น๐ฒ ๐ฏ๐ฎ๐ฐ๐ธ๐ฒ๐ป๐ฑ๐
Think: Qdrant, Pinecone, Weaviate, Milvus, etc.
No more bottlenecks๐
โ
File parsing: now plugs into any ๐๐น๐ฎ๐บ๐ฎ๐๐ป๐ฑ๐ฒ๐
-๐ฐ๐ผ๐บ๐ฝ๐ฎ๐๐ถ๐ฏ๐น๐ฒ ๐ฑ๐ฎ๐๐ฎ ๐น๐ผ๐ฎ๐ฑ๐ฒ๐ฟ
Using LlamaParse, Docling or your own setup? Youโre covered.
Curious of knowing more? Try it out! ๐ https://github.com/AstraBert/ingest-anything
Organizations
