The What, Where and Why of Mantabase
Hey there -
I'm Nicolas, one of the co-founders of Mantabase. For our first blog post, I wanted to share our backstory.
We are, as they say, "tech people". Our team met about a decade ago while working for a (at the time) tiny startup in France. We sat side by side at large desks, punching at our computers at all hours until we were able to ship code we felt proud of and could jump to the next feature.
Over the years, we've hopped from one company to the next, grown some teams to unmanageable sizes, started a few side projects, and generally enjoyed coding and building products. I guess we're probably addicted to the feeling of pride we get when people enjoy using a product we built.
Nothing is more satisfying than seeing the joy on someone's face as they read the documentation of Stripe, build a crazy fast search with Algolia, or monitor a large infrastructure seamlessly with Datadog. We've seen these smiles before, and we want to see more!
Across the years, as an all-too regular part of our jobs, we’ve been faced with the task of having to build a crawler. Most devs do at some point or another, it feels like. Scraping a small sample dataset to build a customer demo, automating a pipeline that connects a website's content to an API, engineering a real-time tool to automatically align the pricing of a product based on the competition’s pricing... Crawlers are everywhere.
And crawlers are... tricky. They are the type of projects that start with "I'll be done in a day", and ends in nightmares a couple years later when the pipeline goes down for the 3rd time that week. Sounds familiar?
There's been many great attempts from the industry to make crawling simple, but… we're not there yet. Ask any developer who has coded and maintained a large crawler in production, and they'll tell you how painful it is. The number of edge-cases, boilerplate code, tedious fine-tuning, gluing, stitching, and maintenance issues that developers face are - frankly - too much. Even seasoned devs struggle with it.
We looked around and tried a few solutions, from a pure wysiwyg to a variety of complex dev frameworks and they all simply didn't cut it. They didn't make crawling simple. Particularly at scale. Particularly in production.
We ended up hating having to build a crawler, sometimes voluntarily avoiding these projects until they inevitably fell into our lap. We knew something had to be done. There had to be a better way. Crawling isn't anything insanely complex. You tell a piece of software what to do on a website and it does it. “Put this dataset in this bucket once a day please, thank you”.
Mantabase is our attempt at doing this. Build the product we wanted to use throughout all these years. Scratch our own itch. And hopefully help others, too. Our goal is to make you love building crawlers again.
We know that we have a lot of work ahead to get there, and we're committed to giving it our best. But most of all, we feel a deep sense of responsibility to make a product you can rely on, and one you love using. And if we're lucky, we'll be proud of what we built.
