Hi Alvin users and fans,

We hope you had a fantastic July! Now that the dust has settled, we wanted to take a moment to reflect on the exciting updates and enhancements we brought to Alvin during July.

New Features & Improvements

Lineage layers

We are very excited to announce layers in the lineage graph explorer. It’s been a long time coming!  The first layers implemented are volume and usage. Read more about it here!

Data volume anomaly detection

We have just released the first one of our key general data observability features: Data volume anomaly detection. It works by regularly snapshotting row counts for a table and applying an ARIMA model to detect anomalous changes based on the last 30 days. An interesting note is that we did a lot of iterations on this with our customers, with the assumption that using more granular job data such as rows deleted/inserted/updated etc would yield more precise results. The experiences showed that simply snapshotting row counts over time produced more stable results in terms actually catching anomalies.

Freshness anomaly detection

Another observability stable - data freshness anomaly detection has been rolled out. It’s as simple as you would expect - by comparing the update times of a table over time to the most recent expected update time we notify when something should have been updated is not.

Behind the Scenes: Tackling Bugs & Refining the Experience

We've also dedicated time to squashing pesky bugs and fine-tuning Alvin to ensure a smoother, more enjoyable experience for everyone. Some key accomplishments include:

  • UI/UX improvements: Whenever you take actions in the UI such as adding filters etc, URLs are always reflecting the view state. URLs across the product are thus shareable which makes the experience of sharing insights with coworkers on i.e slack a lot easier! In addition, easy search and go-to node in the lineage graph has been added.
  • Performance improvements: Some refactoring in the backend code improves caching and waiting times, making the overall experience snappier!

On the Horizon: What's Next for Alvin

We're always looking forward, and here's a sneak peek at what we're working on for the coming months:

  • Gcp projects workloads and anomalies/alerting: A requested feature from BigQuery users has been cost anomaly detection on a GCP project level, as this is often associated with different teams or domains.
  • Service account workloads and anomalies/alerting: A similarly requested feature from BigQuery users has been cost anomaly detection on a service account project level, as this is often associated with different services/functionality like Airflow, dbt or other more company/domain specific workloads.
  • Anomaly grouping: Some anomalies come in groups, it’s easy to understand when you look at a lineage lineage, an anomaly in volume would often “trickle down” the graph and cause a cascade of anomalies - but the advantage is that lineage can be used to detect this and present it more concisely! 
  • Some more easter eggs! Stay tuned!

Your Feedback Matters to Us!

We're committed to continuously improving Alvin, and your feedback is essential to our progress. We'd love to hear your thoughts and suggestions on the recent updates, or any other ideas you have. Just let us know by replying to this email or reach out on LinkedIn or Slack.

Thank you for being a part of our journey!

The Alvin Team

‍