When You Are With the Right Organization You Don't Need to Wait on Fresh Pivots

“People looked at me differently.  They knew I was with somebody. 

I didn’t have to wait in line at the bakery on Sunday morning for fresh pivots.”

-Henry “DAX” Hill

 

 
That Second Refresh Can Really Bite You

Back before the great distraction known as Donald Farmer hijacked this blog for a few days, I was talking about Scheduled Data Refresh.  Specifically, I left off talking about how refresh is really TWO refreshes – PowerPivot and Excel.  Here’s the illustration again:

powerpivotrefreshvsexcelrefresh

OK, so the PowerPivot refresh service has one primary mission in life, and that is to refresh the PowerPivot model.  Which brings us to…

Surprise #1: PowerPivot refresh service does NOT refresh the Pivots!

That’s right, it leaves the pivots in the workbook alone.  So by default, those pivots (and/or cube formulas) still contain stale data!  I’m 100% serious.

“But wait, Rob, you’re wrong!” you say.  “I’ve tested this feature out, and I have NEVER seen stale data in my pivots when I view them in Excel Services!”

That’s right, you DON’T see stale data do you?  That PowerPivot refresh service is one resourceful beast, and it’s playing a clever little trick…

Surprise #2:  PowerPivot refresh service sets “Refresh on Open”

Have you ever seen this feature in Excel?  It’s buried pretty deep:

Refresh On Open Setting in Excel is Used By PowerPivot Scheduled Refresh 
Refresh on Open Setting in Excel 

Yep, click Connections on the ribbon, select a connection (typically named “PowerPivot Data” in our case), then click Properties.  On the resulting dialog you will see the checkbox “Refresh data when opening the file.”

By default that checkbox is NOT set.  You can try this out to see what I’m talking about:  take a PowerPivot workbook, open in Excel client and verify the checkbox is not set.  Then upload to SharePoint, schedule a refresh.  When complete, download the workbook and look at this setting again.  It will be checked now.

Side Topic:  How Does PowerPivot Refresh DO THAT???

I’m 99% certain that there is no API on the server for doing this.  I don’t think Excel Services “helps” PowerPivot at all.  I’m pretty sure PowerPivot modifies the workbook directly, via the file format.  The Open XML File Format, to be precise…  which happens to be what Office uses – XLSX, DOCX, PPTX – these are all Open XML files.

You can do pretty astounding things with that format, so if you are a developer type, I suggest playing around with it.  I can tell you that OUR developers at Pivotstream had a heart attack when they saw that SDK 🙂

How This Can Bite You:  Refresh Sometimes Takes Awhile

HOURGLASS So far so good.  PowerPivot refresh does NOT refresh the pivots, but it DOES set the refresh on open flag.  So when you open it in Excel Services, the first thing Excel Services does, before it shows you any workbook content, is refresh the pivots.  So you never see stale data.  Ever.

In many cases, that’s the end of the story, everything is happy.  But in other cases, it can be painful.

Remember the post I did awhile back on the differences between Update and Refresh?  The overall theme of that blog post was basically:  Refresh Can Be Very Slow.

Why is that?  Short version:  Refresh refreshes EVERY pivot in the workbook.  AND refresh does more work than a normal pivot query.

So…  if you’ve got a lot of pivots in your workbook, or a large data model, or complex measures, or perhaps a combination of those, refresh can take awhile.

Other Considerations

The worst part, of course, is that when a user opens a report in the browser, they have to wait awhile before they see any data.  Sometimes that’s just an extra second or so, which is why you may not have noticed.

In other cases though, it can be as much as 1-2 minutes!

That happens for every user.  In fact, if you open a browser and hit a report, wait through the refresh, and then close the browser, even if you immediately return to the report in a new browser window…  you have to wait out the refresh again.

Imagine what that can do to the CPU(s) on the PowerPivot server(s).  Of course, if you planned your hardware around data refresh as the peak load scenario, then you likely have enough CPU, but still…  a lot of users doing this at once will only magnify their wait time.

“Give me the good news!”

First of all, I brought this to the attention of my former colleagues at Microsoft last time I was in Seattle.  They’re working on solutions that don’t involve “Refresh on Open” but it’s not yet known when or how those solutions will be made available.  Kudos to them for their responsiveness on the matter – I will keep you posted.

In the meantime, there are obvious things you can experiment with.  Keeping the number of pivots smaller is one.  Cutting back on cross-filtering in slicers is another.  But fundamentally, if you have a reporting/modeling/analysis scenario that involves heavy lifting, you can’t exactly just remove those needs from your workbook.

At Pivotstream, we found ourselves in precisely that situation.  And I am happy to say that we have beaten the problem, but the answer is complex – deserving of its own series of posts in fact.  Plus, there are some things here that I am not yet ready to talk about publicly, muhaha…  so let’s put that on the back burner for now.

Next up in this series:  “thin” and “core” workbooks, aka, “hub and spoke.”

Click Here for Part Three >>