Hi Alvin users and fans,

We hope you had a fantastic June! Now that the dust has settled, we wanted to take a moment to reflect on the exciting updates and enhancements we brought to Alvin during June.

New Features & Improvements

Workload Insights

We have just unleashed our SQL analysis capabilities into workloads. This will combine, where available, engine-specific insights with the SQL-level insights. These will continue to grow and evolve, but a few as show below are “SELECT *” and using pattern filters with leading wildcards.

What is really cool is that the when inspecting a workload the specific line/columns from a potentially large SQL statement is extracted and shown, whereas you can look at the query as a whole by clicking further.

We are excited to bring this unique feature forward after seeing many data teams struggling with cost, poorly performing queries and slow dashboards. Although we cannot peek into the query engines just yet, there is often a correlation between poorly performing queries and patterns that are discovered. Some patterns, like “Select *” may be “fixed” by the query optimizer but the question that still stands is how nice it is for someone to read a query without well-defined columns.

Workload filters and groupings

Worklads can now be filtered or grouped by users, domains and teams as well as additional metrics

‍

Workload metrics

We have now added an additional set of metrics for workloads - average coast, number of queries and total duration - with more to come. Stay tuned as we continue to develop features around monitoring and alerting for these metrics.

Storage Insights

One of the key messages from demos, customers and practitioners in the space is table/model explosion, duplication and too many tables. While there are many root causes and reasons for there to be too many tables around we dont’ believe in pointing fingers (and that our customers should also not do it internally). But it’s important to have transparency into what is being used and not, and where tables fit into the data environment; are they purely part of a big pipeline or do they have analytical usage associated with them and so forth. 

We are super happy to announce the release of our storage insights that merges together storage cost, processing cost as well as semantic usage analysis to pin-point tables that should be deleted, investigated or warrants further inspection. In many ways it’s a simple feature but also a culmination of our core capabilities to process SQL and metadata into a clear UX. Like with workloads, storage can also be broken down and filtered by numerous dimensions such as table database, schema, owner, domain and team.

Tables are, based on lineage and usage information, automatically classified, such as unused tables, source tables, intermediate or leaf tables. This makes it very easy to reason about tables and take appropriate actions confidently.

Behind the Scenes: Tackling Bugs & Refining the Experience

We've also dedicated time to squashing pesky bugs and fine-tuning Alvin to ensure a smoother, more enjoyable experience for everyone. Some key accomplishments include:

  • Improvements to the parser: Supporting numerous new SQL dialect additions to Snowflake and BigQuery in the wake of recent announcements
  • Grouping improvements for tables: Now supporting dbt’s Slim CI (schema names).

On the Horizon: What's Next for Alvin

We're always looking forward, and here's a sneak peek at what we're working on for the coming months:

  • Observability: Get ready for automatic, ML-based anomaly detection of issues related to timeliness of jobs and row counts (and possibly some easter eggs)
  • Lineage performance improvements: As we are seeing more advanced use cases for lineage, we are also investing in making it ever more performant to make it work at scale!

Your Feedback Matters to Us!

We're committed to continuously improving Alvin, and your feedback is essential to our progress. We'd love to hear your thoughts and suggestions on the recent updates, or any other ideas you have. Just let us know by replying to this email or reach out on LinkedIn or Slack.

Thank you for being a part of our journey!

The Alvin Team