Engineering solutions for the data-driven world

Unlocking Data Potential: The Shift from Manual to Automated Workflows

When I first arrived at the company, I didn’t step into some sleek, modern data operation. Like many growing businesses, data lived in Excel silos — scattered across desktops, tucked away behind an on-prem VPN, or emailed around as attachments.

Reporting often felt like a scavenger hunt. If someone asked for a metric, it usually meant tracking down the “right” file (which wasn’t always the same depending on who you asked), exporting CSVs from SaaS platforms, and patching everything together with fragile formulas. Half the time something broke, and the fix depended on whoever originally built the file.

It wasn’t anyone’s fault — the systems just hadn’t caught up with the business needs yet. But it was inefficient and fragile.

At the time, my main tool was Excel. And I’ll admit, I pushed it hard. I built dashboards that admins loved and analyses that actually helped decisions get made. On the surface, it looked like I was adding value. But behind the scenes, it took serious effort to keep the wheels turning.

Just preparing one month’s worth of data took around 30 hours of tedious copy-pasting, formatting, and cleaning before I could even begin analyzing. By the time the numbers were ready, the files themselves were bloated, fragile monsters that could collapse with the wrong keystroke.


The First Breakthrough

Out of necessity, I started teaching myself Python. At first, it felt like a foreign language. But then I rewrote my first big workflow in code — and something clicked.

That same process that used to eat up 30 hours of my month? My script ran it in seconds.

Seconds.

It was one of those career-changing moments. Suddenly, I wasn’t stuck grinding through the same painful process on repeat. I had built something that could do it for me.

The real turning point came a few weeks later, when I discovered something hidden in plain sight: the SaaS platform I’d been manually exporting CSVs from actually had a developer portal with open APIs. I didn’t even know what an API was at the time. But once I figured out how to use it, the game changed completely.

No more clicking menus. No more CSV exports. I could pull clean data, on demand, whenever I wanted.

That lesson — learned only a few months into my first corporate job — permanently rewired how I thought about work. Every repetitive task suddenly looked like an opportunity for automation. Every manual process looked like wasted potential.


A Bigger Door Opens

Around that same time, the company signed a contract for the Microsoft 365 E3 suite. To most people, it was just another IT update. To me, it was an open door.

I signed up for a Power BI free trial and started experimenting. At first, I connected it to the same local Excel files I had been working with. But then I realized something huge: Power BI could connect directly to SharePoint.

That meant I could publish dashboards that anyone could access without emailing giant spreadsheets around. So I created my own SharePoint site and started using it as the foundation of a centralized reporting system. It wasn’t pretty, but it worked.


Closing the Loop

Even then, I still found myself dragging API-pulled files into SharePoint manually. And every time I did, I thought: this feels wrong.

If I could automate the extraction, why not the upload?

That’s when I discovered the Microsoft Graph API. Suddenly, I could script the whole loop:

  • Python pulled data from SaaS APIs.
  • Pandas cleaned and transformed it.
  • Graph API pushed it straight into SharePoint.
  • Power BI picked it up for dashboards.

No more manual steps. Data was flowing end-to-end without me touching a thing.


Beyond Reporting: Driving Action

At that point, the dashboards were helpful — but I started asking myself: why stop at reporting?

If the real goal was to help people act faster, why not make the reports themselves the starting point for action?

So I dug deeper. By inspecting browser developer tools, I learned how to uncover deeplinks buried inside our SaaS platforms. With some string interpolation, I could dynamically generate links in reports that sent people directly to the exact item they needed to act on.

When I showed this to my boss, his jaw dropped. “How did you even figure that out?” he asked. To me, it was just tinkering. To him, it looked like magic.

That’s when I realized — sometimes the simplest tricks can have the biggest impact.


The Scrappy Architecture That Emerged

By the time it all came together, here’s what the system looked like:

  • Data sources feeding in through APIs.
  • Python scripts doing the heavy lifting on transformation.
  • SharePoint acting as a makeshift data warehouse.
  • Power BI delivering interactive dashboards.
  • Power Automate handling report distribution and Outlook communications.
  • Deeplinks embedded so reports weren’t just about what was happening, but also where to fix it.

It was scrappy, yes. And SharePoint had real limitations — slow reads/writes, no indexing, scaling headaches. In a perfect world, a proper data warehouse would’ve been better.

But for where we were? It was a massive leap forward.


Lessons from the Lone Wolf Phase

Being the only data person forced me to learn some lessons the hard way:

  • Document everything. If I got hit by a bus, nobody else would know how the pipelines worked.
  • Scrappiness is a superpower. I didn’t wait for perfect tools or formal training — I just used what I had.
  • Adoption matters more than elegance. A perfectly architected system nobody uses is worthless. A messy one that people rely on changes the business.

Looking Back

When I think about that journey — from Excel headaches to automated pipelines and action-driven dashboards — I don’t really see it as a story about tools.

The real story was about a mindset shift. I stopped seeing data as something to manage and started seeing it as something to leverage. I stopped asking “how do I get through this process?” and started asking “why am I doing this process at all?”

The biggest impact wasn’t in any single script or dashboard. It was in building a system that made data reliable, accessible, and actionable. A system that turned reporting from a weekly chore into a real asset.

And it all started with one 30-hour Excel headache — the kind of pain point that, if you lean into it, can change the trajectory of an entire career.

Leave a comment