June 2020 was a significant month in the history of Carguero, a Brazilian startup transforming the cargo transport market.

Before then, all of Carguero’s dashboards were filled directly from their SQL Server (gasp!). Imagine all this raw data, entirely unsuitable for the eyes and brains of a business person looking for actionable insights. The data was slow and cumbersome to consume, defeating the purpose of self-serve analytics.

Data should empower you to ask questions about your business — and find the answers. If it’s too hard to even ask the right questions, there’s probably something wrong.

Everything changed when they hired Thiago, their first data person. His trained eye, having been immersed in the data world for years, quickly spotted that the company’s rudimentary system needed to change.

He realized that, at first, what Carguero needed was a data lake and/or data warehouse to store analytical data for dashboards and the BI team to consume. After some deliberation, they ended up with a data lakehouse solution, a popular hybrid mode for storing information.

What may seem like a small change for some greatly impacted the company. But that, dear reader, was just the beginning: it was time for Carguero to become a real data-driven company.

Starting a data revolution

Being genuinely data-driven means digging deeper into all your raw data, refining it, and increasing profitability based on what you find.

That takes a lot more than just dumping all of your data into an analytical database. You need to define processes, bring in the right people, and get the entire company’s buy-in. Everyone needs to be on the same page about the value of high-quality data.

Thiago and the rest of Carguero’s data team knew that to get what they needed to turn data into actionable insights, they needed everyone to get it. So they stayed 100% on brand and used data… to prove the value of data. One spreadsheet at a time, he and his team convinced every stakeholder to hop on the data boat.

Now they just needed the right tools.

The search for an end-to-end data observability tool

With the resources and freedom to invest in data, it was time for the team to find their tools. It wasn’t long before they realized a data observability tool was a must if they wanted to take this data-driven thing seriously.

Because their BigQuery data lakehouse was being fed by…

  • SQL Server
  • Hubspot
  • Freshdesk
  • Jira
  • 55pabx
  • Mongodb
  • Postgres
  • Feedz
  • Docz

That’s a lot of sources, each with its own structures and ways of storing data. So where was all their information coming from? And where was it going? Losing track was all too easy. No pressure then on finding the right tool to monitor all of this, right?

Right?

Spoiler: The wrong data monitoring tool is all too likely to pile more work on everyone’s plate, so this was a bit of a make-or-break moment. Rebeca, one of Carguero’s data engineers, was the lucky one to get to test out all the tools that looked promising, including Alvin.

And how did she measure which tool would be the best? You’ve probably spotted the pattern here: She did it with data, of course.

Rebeca ended up with a spreadsheet that broke down all the tools by Carguero’s most important criteria:

  • Time to implement
  • Need for constant fixes/adjustments from the team
  • Monthly value
  • Visibility to the rest of the company
  • Total cost
  • And of course, the amount of people involved for the tool to work

With Carguero’s most important use cases being lineage and impact analysis, Alvin came out on top (the spreadsheet doesn’t lie) and the team decided to give it a go.

And they haven’t looked back since.

How Carguero uses Alvin

“Alvin helps us with two very important pillars of data observability principles: schema and lineage,” says Thiago. “We know exactly what will happen if we change something in the data, we know what will break in the pipeline, and why.”

“The lineage helps us to have an end-to-end visual of the table, that is, how it was created to where it is being used, all at the column level,” says Rebeca, the one who had mathematically proven that Alvin was the best fit for Carguero.

“Impact analysis helps us to verify if a change in a certain column/table is possible, or if it would impact other tables or even jobs”, she continues, “which is very powerful.”

Team Alvin is proud to have played a part in enabling another company’s deepening addiction to better data, and we’d love to do the same for you.

‍