I sat down with the billing team to figure out if Microsoft Copilot could actually help with the mess we’ve been dealing with since switching to our new billing system. Spoiler alert: it was equal parts promising and frustrating, and required two separate sessions to even get close to something functional.

The Real Problem We’re Trying to Solve

Let’s be honest about where we are right now. Our new billing system has bugs. Not minor bugs—the kind that force us to create Excel workarounds for things that used to just work in our old system. The foreign agent and foreign insured reconciliation reports that the team needs to produce? They’re taking longer and requiring more manual intervention than they did before the “upgrade.”

The company has been working on building reports directly from the Billing 360 database to work around some of these issues, but in the meantime, the team is stuck doing more manual work than anyone signed up for. That’s the context we’re operating in.

What We Actually Tried to Do

The training session started with the basic premise that Copilot works best when you treat it like you’re training a new employee. You need to be specific about what you want, where the data should come from, what format you need, and all those details.

We’d already had some success getting Copilot to generate a one-off email summary PDF that analyzed our billing email patterns. That worked. So the natural next step was: can we make this happen automatically every month instead of manually recreating it?

The plan was to build a custom Copilot agent—basically a saved set of instructions that someone could just activate and say “generate this month’s report” and get a PDF out the other end.

The Excel Breakthrough Moment

Before we even got to the automation attempts, we had a moment that actually sold the team on the potential here. We took one of those Excel reconciliation reports—the ones that currently require manual review and analysis—and uploaded it to Copilot.

The results were genuinely promising. Copilot was able to analyze the data, identify patterns, flag discrepancies, and present it in a way that would normally take someone time to compile manually. When the team member saw what it could actually do with real billing data, you could see the light bulb go on.

This wasn’t theoretical anymore. This was “oh, this could actually help me with the daily work I’m stuck doing because of the new system’s limitations.” That Excel exercise is what made the automation attempt worth pursuing—we’d seen it work with actual data.

Where Things Got Real (Session One)

This is where the training session got interesting, and by interesting I mean we hit some walls.

We created the agent with what we thought were clear instructions. Here’s what we were working with:

Initial Agent Prompt: “Generate a broad email summary for billing within the last month. Provide an email summary for the month and provide downloadable PDF version.”

Simple enough, right? We then tried to test it with: “Generate report for the month of December.”

Except when we tested it, Copilot came back saying it didn’t have direct access to Outlook email data. So we tried to be more specific:

Revised Instructions: “Grab email data from Outlook. Billing account. Grab actual email data.”

After trying various approaches, we discovered the core issue: the standalone Copilot app couldn’t access email data directly. We had to return to the original conversation within Outlook’s Copilot interface to make it work.

But even that solution proved unreliable. Running the same report multiple times produced inconsistent results—sometimes pulling only a week’s worth of data, other times reporting different email totals. This inconsistency undermined our confidence in using it as a dependable automation tool.

The first session ended with us realizing we needed to resolve fundamental access and permissions issues before proceeding. Not exactly the triumphant conclusion we’d hoped for, but a necessary reality check.

Round Two: Actually Making It Work

After addressing the technical access challenges, we regrouped for a second attempt. This time, we made real progress.

The second session focused less on concepts and more on practical implementation. We refined our prompts, clarified data sources, and systematically worked through the permissions and connections required for Copilot to actually retrieve email data.

Challenges remained—we still encountered unexpected behaviors that required troubleshooting. But we shifted from “this doesn’t work at all” to “this could actually be useful with some fine-tuning.”

The crucial insight: Copilot requires extreme specificity. You must spell out which account, what date range, which format, what to include, and what to exclude. Think of it as training a very literal employee who will execute your instructions exactly as stated—no interpretation, no filling in gaps.

The Prompts That Actually Mattered

Through trial and error across both sessions, we learned that effective prompts needed to include:

• Specific data source: Not just “email data” but “email data from the billing Outlook account”
• Clear time frame: “For the month of December 2025” rather than “recent” or “this month”
• Exact output format: “Generate a PDF summary report” not just “summarize”
• Confirmation of access: Actually confirming that Copilot has permission to access the data sources before trying to run the automation

The difference between a prompt that fails and one that works often came down to these specific details. Frustrating? Yes. But also useful to understand for future automation attempts.

What We Actually Learned

Despite the technical hiccups and need for multiple sessions, there were some genuine insights:

• Excel analysis works really well: Uploading actual billing data and having Copilot analyze it produced legitimately useful results. This isn’t just theory—it helped with real work.
• The specificity thing is real: You really do have to spell everything out for Copilot. It’s not intuitive like a human would be. If you assume it knows what you mean, you’ll be disappointed.
• Access and permissions matter: Before you can automate anything, you need to make sure Copilot actually has access to the data sources you’re trying to use. This isn’t always straightforward and may require separate setup work.
• Iteration is the norm: The first version of what you build probably won’t work perfectly. That’s expected. You refine it based on what breaks. Sometimes you need multiple sessions to get it right.
• The potential is there: Once we worked through the technical access issues, the concept proved sound. This could genuinely save time on repetitive reporting.
• You’re training a very literal employee: The comparison to training a new employee is accurate, but specifically a new employee who will do exactly what you say with zero interpretation or common sense applied.

The Honest Assessment

Is Copilot going to magically solve all our billing system problems? No. The new system still has bugs, and those need to be fixed at the source.

But could it help reduce some of the manual burden of creating routine reports while we’re dealing with those larger system issues? Yes, actually. The Excel analysis alone proved that. Once we worked through the technical setup challenges across both sessions for the automation piece, we got to something that could legitimately save time.

The value isn’t in replacing human expertise or dramatically transforming how we work overnight. It’s in potentially automating specific, repetitive tasks that currently eat up time—like analyzing Excel reconciliation reports, compiling monthly email summaries, or generating consistent reports from the same data sources.

With our new billing system being slower than the old one, any legitimate time savings on routine tasks is meaningful. Not revolutionary, just helpful. And given that the team is already dealing with more manual workarounds than they should be, even small wins matter.

Where We Go From Here

The next steps are practical:

1. Continue using Copilot for Excel data analysis—this already works and provides value
2. Document the exact prompts and setup that actually worked for automation so others can replicate it
3. Test the agent with real monthly data to make sure it’s consistently reliable
4. Identify other similarly repetitive tasks that might be good candidates for this approach
5. Be realistic about what’s worth automating versus what needs human judgment
6. Keep refining based on what works and what doesn’t in actual use

This isn’t a story about AI transformation or revolutionary change. It’s about a billing team trying to claw back some productivity while dealing with a buggy new system, and exploring whether AI tools can provide some tactical relief.

It took two sessions to get the automation piece somewhere useful. There were frustrating moments. But between the Excel analysis that already helps and the automation that’s getting close, we’re building something that might actually make the daily grind a bit more manageable.

Some things will work, some won’t. But at least we’re trying practical solutions rather than just complaining about the problems. And sometimes that means sitting through two training sessions to figure out how to make the robot do what you want.