Almost last article in the series. This article will focus on last Power BI tests of Copilot. In next (and last) article, I will try to summarize all the findings, and set guardrails for responsible adoption of Copilot in your organization. Stay tuned.
Similarly to what you could see in previous article, Copilot Pane is available to you at hand, while exploring Power BI Report:
Figure 1. Copilot’s greeting message.
Also, as seen in previous article, one of the options mentioned in the documentation is hidden in the prompt guide:
Figure 2. Prompt guide options.
It kind of makes sense. It is the most detailed option, which could result in more very specific prompts. But there is more. On top of given options, you are also allowed to write custom prompts. These are the options I want to cover in this article. I will be still using my Retail data, as it was in previous article. I will also use the report generated by Copilot, therefore maybe the results will not be that great, but the most important goal here is to assess the cost.
Important NoteCapacity used for testing is P1 (equivalent of F64). If you are using different SKUs, you must scale the results accordingly.
Create bulleted list of insights
Let’s start with first option available.
Figure 3. Bulleted list of insights output – attempt 1.
It took almost 3 minutes to generate the insights. Quite a lot. I wanted to give it a go once more. This time it took less than a minute to generate the output, using “Retry” option. Output was also slightly different:Figure 4. Bulleted list of insights output – attempt 2.
What could we say about the outputs? It looks a bit better than what we could see so far as output generated by Smart Narratives visual in Power BI. If this is your first impression, you nailed. You could even use Smart Narratives visual, which is now powered by Copilot, to include this type of insights as a part of your Power BI Report. Is it really useful? Not sure, seems like a lot of reading, that should rather be replaced with easy-to-understand visualizations. On the other hand, people very often will turn towards text when they are offered a choice. As a report designer, you are probably trying to satisfy a lot of requirements, and this could be one of them.
What I really like about this feature are annotations you may find for each bullet. When you click on them, they will highlight the chart that was used to generate the insights:
Figure 5. Bulleted list of insights output – annotations.
I see here a huge potential to improve data literacy in organizations. You not only receive insights, but they are visibly connected to charts in Power BI Reports. You can use this information to learn how to understand certain visuals, so maybe in future you don’t need Copilot anymore. You will have a look at the report page and immediately get your insights. This aspect changed my mind about using Copilot in this specific scenario. Let’s move to next option.
Give me an executive summary
I think we can expect this option to be much more verbose than bullet points. You can see the result below:
Figure 6. Executive summary generated.
Well, this is not the greatest report we could imagine doing a test of Copilot. Still, let’s try to derive some value from generated output. First, we see short description of the content, which is correct. Surprisingly, Copilot provides incorrect information on time period. There is data available till 30th of November, not September. You could probably expect this type of information to be fairly easy for Copilot to extract. In second paragraph, Copilot picked up a trend, which is quite neat, especially in a report like this one. I can show it using again the same highlight feature as was presented for bullet points:
In this case, my terrible report is actually perfect for Copilot test. If you ever see a visual like this, what is the trend you can spot here? Yet, Copilot did that and provided information that there is overall increasing trend of 3.4 %. Also, we see information about steepest recorded incline, which again would be rather hard to spot.
Next information is basically useless, as Copilot found similar trend for TaxAmount, which is rather obvious from business perspective. But important insight here is that Copilot not only is capable of analyzing trends, but it also points out the similarities, which in other case could be very useful information.
Last paragraph is more an implication than another finding. If trends between Profit and TaxAmount are similar, then of course we can expect there is a correlation between these values. Still, in more useful report, being able to immediately spot correlations can be critical. We could argue why not making it in clearer in the report itself, but we are looking at it from beginners’ perspective. After all, even entire report was created with Copilot so far, so let’s assume that these are first steps for someone, and let’s be more understanding 🙂
Overall, it’s quite ok. Nothing that could shock us, but let’s be honest, this is a low-quality report, and was not built following guidelines provided by Microsoft. Just so we don’t leave this area too early, let’s give the executive summary another try, as was done for bullet point insights.
Figure 8. Executive summary generated – second attempt.
I can see two similarities here. Output generated using “Retry” option was much faster than first one. It was again minutes vs seconds kind of difference. As you can see, second output is a bit different as well. Which of course makes sense, why otherwise you would hit the “Retry” button, if you were satisfied with first output. It is more crips than the first one. Is it more useful? Hard to assess with my report, but we can see here majority of overlapping information, presented in slightly different way. We have one more standard option to try.
Summarize visuals on this page
This may be a bit unlucky test. After all there is only 1 page in my report. But this will be interesting to see if there is any significant difference in generated output. This would rather suggest different approach applied by Copilot.
Figure 9. Summarize visuals on this page.
As expected, outputs share lots of similarities. It is slightly different, but it was also the case for second attempt of Executive Summary. The only thing I wonder about here is why this option is a bit hidden. Executive Summary is the one that is exposed together with bullet points insights. For large reports, they are going to be probably quite heavy on Capacity resources. Having only two options highlighted; I think Microsoft could easily fit in there a third one. Not every user will go through documentation, and most probably they won’t even notice there is an option, allowing them to limit the scope of insights generated to a single page of their interest. Maybe this will come in future.
Custom prompts
This feature could be an answer to the problem I highlighted above. After all, as much as exposing some of the options to end user will help get the results they need, being able to chat with Copilot is something people are really looking for. Let’s see if we can simply ask Copilot to summarize specific page of the report.
Figure 10. Custom prompt – summarize report page.
It worked. Summary is given, making me less worried about missing option button as discussed above. I am pretty sure that asking for summary in chat will be most common usage. Again, information provided on date period is a bit messed up. Now it’s even 29th of September for whatever reason. On the other hand, we see the same information on increasing trend of 3.4 %, the same decline period mentioned between August and September. Overall result is pretty aligned with what’ve seen before. Let’s try something else:
Figure 11. Custom prompt – summarize trends.
This time, prompt was a bit more specific, asking to elaborate on trends that can be found on a report page. Considering how simple this report is, we can’t expect any revolution in the output. Rather schematic response from Copilot, with lots of overlapping information. Yet, last paragraph is totally new, mentioning that high profit days repeat in 6 days cycle. This again could be a powerful insight, impossible to get from a terrible report like this one. All we needed to do is ask more detailed question. Before we call it a day, let’s send two more custom prompts:
Figure 12. Custom prompt – September data.
Output is generated, proving, that we can ask Copilot to generate insights on filtered data. Here I asked specifically for September data and generated insights are related only to that period. I like that Copilot doesn’t try to make stuff up, and clearly says that sometimes there is not enough data to say something for sure.
Since Copilot proved to be useful in filtering the data, let’s try to set it on trap, and ask for the time period that doesn’t existing in the dataset:
Figure 13. Custom prompt – incorrect time period.
Phew, that’s a relief. Copilot luckily is smart enough not to assume that end user is right about the data. What is more, end user is persistent and tries to convince Copilot. Luckily it doesn’t work, the answer is strict and correct.
How much does it cost?
Finally, the most important part of this article (at least for me :D). Let’s remind ourselves, that Capacity used for testing is P1 (equivalent of F64). I will also do the math for you and in total I sent 9 prompts to Copilot:
Figure 14. Capacity consumption – last 7 prompts.
Figure 15. Capacity consumption – first 2 prompts.
Total Capacity consumption is 0.86 % for my session. Sending 9 prompts to Copilot during a single session is a lot? Again, hard to say. I can imagine this kind of sessions ending even after first prompt. Especially in case of experienced users. Let’s try to scale it down and assume that standard session will consume not more than 0.5 % of base capacity on average. Now, the other scaling problem is related to number of users sending prompts during 24 hours period. For all the other experiences, that more developer based, I started with calculation for 20 users, but in the end, I assumed concurrency level of 10 users a day. However, report consumers are the largest group of users in organization, assuming that only 10 users a day will utilize Copilot is probably far away from being true. Let’s remember that Copilot can be used starting as of P1 or F64 Capacities. These are large capacities, with potentially hundreds of users visiting reports every day. Having only 30 users a day, will add additional 15 % to total Capacity Consumption. 60 users will add 30 %, and so on. Is it realistic? In my opinion – yes. And let’s not forget, we have more Copilots being used during the day by Developers.
Interesting observation I made is presented on below screenshot:
0.19 % is a prompt to Summarize report page sent with custom prompt. 0.09 % below is the same request sent using built in button available in Copilot dialog window. This is rather huge difference. I wonder whether you receive the same results for that kind of test.
Conclusion
Having in mind, that next article is going to be summary of Copilot tests, I will not include the total number here yet. I will leave it for last article in the series. So far what we could say is that Copilot used on top of Power BI Reports could be a powerful ally for users, who are completely new to analytics, and need support in terms of reading the visuals and deriving value out of available charts. What is more, output generated by Copilot could be even used as a suggestion to report developers, what are additional insights that could be included, or exposed in better way in Power BI Reports.
Only one article left on Copilot topic. There were some changes introduced recently, that I will try to include in my next post.
As always, thank you for reading and see you in next article 🙂
Pawel Wrona
Lead author and founder of the blog | Works as a Power BI Architect in global company | Passionate about Power BI and Microsoft Tech
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.
You can revoke your consent any time using the Revoke consent button.
