PowerPivot database woes

Soldato
Joined
2 Aug 2004
Posts
8,041
Location
Buckinghamshire
I've never been great with database management, but I'm trying to pull together a refreshable standardised report where the end user can select a time frame to look at.

I'm bringing data into PowerPivot by:
  • Exporting data from a server (via Microstrategy in Excel)
  • Manipulating the data via VBA then saving to CSV
  • Import the data via a PowerQuery into PowerPivot (I transform data so I can split one column such as "Value Sales LY" into two "Value Sales" & "LY" - so I can use these as slicers)
The data basically has product sales by month, with each month represented by the first day of the month (eg 01/01/2019 for Jan 2019). There's a column for facts (Unit, volume and value) and another column for if the fact is for this year (TY) or last year.

What I want to be able to do next is have a column that includes (so I can have a slicer that changes the data based on looking at latest full year/MAT & latest QTR):
  • Latest MAT (the latest 12 months, working back from the latest month in the data)
  • Latest QTR (the latest 3 months)

I've tried creating a new table with two columns
  • One column with all the months
  • The second with a descriptor (i.e. 'Latest MAT' against all the months)
However I ran into a many-to-many relationship error as this obviously has duplication with months, e.g. Dec 2020 is part of Latest MAT and Latest QTR


I then tried:
  • Creating a 'selector table' which lists Latest MAT, Latest QTR and Latest Month
  • Then three tables such as Latest MAT with all the months against it, Latest QTR with the last 3 months against it
  • Then created a relationship between the selector and the three tables
This then falls down when I try and link the three tables back to the main data table but get this error:



Database tables here:


Any support much appreciated.
 
The "TIME" column in the CSV contains the date, the data is monthly and each month is represented by the 1st of each month. I.e. 01/01/2020, 01/02/2020.

I could just add in a 'TIME' slicer, but was trying to make this document as simple and unbreakable as possible to stop people getting the wrong insights and date out of it.

Tried Googling this all night but keep hitting brick walls. A lot of tutorials are using PowerBI and/or it goes over my head. Even tried the column add function using the calendar function, but can't even get that to work.
 
I know how you feel mate. I dip into PowerBI and the excel basic version of power query/pivot and it is decent. I managed to find weird bugs in it so understand why you tinkered with the back end data (you used VBA, I used Access to shoehorn some data). I would also try querying the data so it does that into one table for you.

After re-reading it sounds like you need to merge some data (use what you have built then merge it into a new Query).

I dip in to it due to random projects at work but can be months before I return to it which is why I am very rusty.
 
....
However I ran into a many-to-many relationship error as this obviously has duplication with months, e.g. Dec 2020 is part of Latest MAT and Latest QTR
...

Maybe I don't understand what you are trying to do. I don't know why you need extra tables here. You only need one. The CSV.
I would just create two columns in the CSV one MAT and the other QTR. Populated as appropriate.
Then create a power pivot against this now "flat" data.
MAT and QTR would be "different" slicers. One would have MAT/NULL the other QTR/NULL, the data will overlap. Since it exists in both

Personally
I'd prefer there was slicer for year then month. I often actually create a column with Month (February etc) and another Year (2021) and have slicers on them.
Let them select the dates themselves.

Because MAT or QTR will change every month, are they really needed?
 
Any support much appreciated.

This query is right up my street, I'm a former MicroStrategy developer/architect, PowerBI/Qlik/Salesforce/Tableu third line support and an ETL programmer.

Power BI used to not detect dates and hierarchy them into year, month quarter, date etc.
Back in the day most people would keep a good advanced query to generate a big comprehensive date table, then relationship the source into it and use the date table for all time slicers.
Excel and Power BI are pretty much the same in the query editor and relationships so the approach works in excel too.

Grab an old date table script from the net, one where you define an earliest date and final and it populates all possible dates.
This date list would also have columns to define the date in all possibilities like month name, month no, day name, day no, year, year-month, quarter.
Etc. Relate your source table into this and then use the date table as a slicer.

Edit:
This page looks decent
https://blog.enterprisedna.co/how-to-create-a-detailed-date-table-in-power-bi-fast/

I'm unsure of your MAT and QTR requirements and what this means but if you can work them into a date table column using a case/conditional statement then jobs good.
If this is some kind of filtered start-end accumulative total count then it's complex but could involve something like a dax column insert with some nice formula that will take ages to work out.

You're reinventing the wheel a bit though. You're using a federated enterprise grade reporting platform (MicroStrategy) that has data control and top tier ETL capabilities then exporting it to excel to use as an interactive report.
Excel is good for small report data sets when you need to power query for some simple ETL without comitting to a BI platform but you should use the excel data in a BI platform for user interactivity.
This will give nice slicers, aesthetics and lock them out from breaking things.


Second edit.
Read the question again.

  • Latest MAT (the latest 12 months, working back from the latest month in the data)
  • Latest QTR (the latest 3 months)

You're using visual basic in excel to transform the data?
I once did that and I'd recommend against it, when you get promoted you can be hounded a bit to fix anything overly proprietary.
Power query is all standardised functions whereas VBA can be proper coding.

MAT
If the 12 months is rolling (ie 12 months from current month) then get the current date and evaluate each row's date and apply values in a new conditonal column.

QTR
Same type of thing, define the months in Quarter if different to standard financial definitions then create a column.

There's many ways to skin a cat, do this in vba or transform it in the query editor (DateTime.LocalNow)

You could also do without a date table as someone mentioned above and do this all in one flat de-normalised table. I'd still recommend a date table though as its a good practice to denormalise things that are absolute like dates and may apply if you ever need to integrate a second data export for a different thing in a dashboard.

If you're looking for a neat visually appealing and interactive plaything for showing off data then try Power BI. It's essentially excel mixed with powerpoint and you can export a multi dashboard page PDF with mixed reports and narrative elements. Use power query to do the data transformations, even though Power BI can run everything verbatim from the power query it's good to keep this separate otherwise you need to change multiple reports every time you modify the data structure.
 
Last edited:
If you're looking for a neat visually appealing and interactive plaything for showing off data then try Power BI. It's essentially excel mixed with powerpoint and you can export a multi dashboard page PDF with mixed reports and narrative elements. Use power query to do the data transformations, even though Power BI can run everything verbatim from the power query it's good to keep this separate otherwise you need to change multiple reports every time you modify the data structure.

Thanks for the post Dude. You sound like a great source of information and experience!

Have to ask on this part, when I had to get stuck into BI, you could download your own personal copy of BI and use it. However when it comes to sharing the data and the like, for business they need a place to run it from like a Sharepoint or Azure cloud space?

This made me u-turn on a project once as I could not get the organisation to provide a place to get the users to view the data. I reverted back to the Power Views and query/pivot but found this lacked a lot of features (it was on 2016 which may have changed now with 2019/365 versions). I like to brush up on this information as I also tend to be half consultant, half provisioner lately in my work and knowing the limitations and better options is certainly useful to stop wasting time and resources.
 
We use SharePoint online, PowerBi Server and Tableau Server, SSRS and a few 3rd party plugins for our website's. We also export to excel if the end users want it in that format. We also have staging and transformation servers. We do lot of the work in SQL and SSIS.

When I was a contractor you end up doing a lot in VBA as that's often all they had. I'd you are a developer VBA is quite powerful. You can write whole systems in it. Though people can be a bit snobby about it.
 
Really appreciate the support guys and taking time to reply.

I've addressed your points below.

Maybe I don't understand what you are trying to do. I don't know why you need extra tables here. You only need one. The CSV.
I would just create two columns in the CSV one MAT and the other QTR. Populated as appropriate.
Then create a power pivot against this now "flat" data.
MAT and QTR would be "different" slicers. One would have MAT/NULL the other QTR/NULL, the data will overlap. Since it exists in both

Personally
I'd prefer there was slicer for year then month. I often actually create a column with Month (February etc) and another Year (2021) and have slicers on them.
Let them select the dates themselves.

Because MAT or QTR will change every month, are they really needed?

But if I create columns then I can't use a slicer as slicers show choices within one column/field.

Whilst I see what you're saying about two slicers, a slicer for each month would be cumbersome because you will need to select multiple items in two slicers. For example, the MAT based on January 2021 would require selecting 2020 & 2021, with month containing a multiple selection of Feb through to January.

All I can say is trust me when I know the limitations of my colleague's ability to translate MAT into selecting the right slicers.

This query is right up my street, I'm a former MicroStrategy developer/architect, PowerBI/Qlik/Salesforce/Tableu third line support and an ETL programmer.

Power BI used to not detect dates and hierarchy them into year, month quarter, date etc.
Back in the day most people would keep a good advanced query to generate a big comprehensive date table, then relationship the source into it and use the date table for all time slicers.
Excel and Power BI are pretty much the same in the query editor and relationships so the approach works in excel too.

Grab an old date table script from the net, one where you define an earliest date and final and it populates all possible dates.
This date list would also have columns to define the date in all possibilities like month name, month no, day name, day no, year, year-month, quarter.
Etc. Relate your source table into this and then use the date table as a slicer.

Edit:
This page looks decent
https://blog.enterprisedna.co/how-to-create-a-detailed-date-table-in-power-bi-fast/

I'm unsure of your MAT and QTR requirements and what this means but if you can work them into a date table column using a case/conditional statement then jobs good.
If this is some kind of filtered start-end accumulative total count then it's complex but could involve something like a dax column insert with some nice formula that will take ages to work out.

You're reinventing the wheel a bit though. You're using a federated enterprise grade reporting platform (MicroStrategy) that has data control and top tier ETL capabilities then exporting it to excel to use as an interactive report.
Excel is good for small report data sets when you need to power query for some simple ETL without comitting to a BI platform but you should use the excel data in a BI platform for user interactivity.
This will give nice slicers, aesthetics and lock them out from breaking things.


Second edit.
Read the question again.



You're using visual basic in excel to transform the data?
I once did that and I'd recommend against it, when you get promoted you can be hounded a bit to fix anything overly proprietary.
Power query is all standardised functions whereas VBA can be proper coding.

MAT
If the 12 months is rolling (ie 12 months from current month) then get the current date and evaluate each row's date and apply values in a new conditonal column.

QTR
Same type of thing, define the months in Quarter if different to standard financial definitions then create a column.

There's many ways to skin a cat, do this in vba or transform it in the query editor (DateTime.LocalNow)

You could also do without a date table as someone mentioned above and do this all in one flat de-normalised table. I'd still recommend a date table though as its a good practice to denormalise things that are absolute like dates and may apply if you ever need to integrate a second data export for a different thing in a dashboard.

If you're looking for a neat visually appealing and interactive plaything for showing off data then try Power BI. It's essentially excel mixed with powerpoint and you can export a multi dashboard page PDF with mixed reports and narrative elements. Use power query to do the data transformations, even though Power BI can run everything verbatim from the power query it's good to keep this separate otherwise you need to change multiple reports every time you modify the data structure.

Just to address some of your points:
PowerBI - Unfortunately I don't think we use PowerBI - it's not on my machine and not on our Software Centre. We use Office 365 on semi-annual Enterprise and the icons under 'this product contains' show the typical Office applications such as Word, Excel, PowerPoint, Outlook, OneNote, Publisher, Access and Skype for Business.

VBA - I know some of what I'm doing in VBA can be achieved during the PowerQuery or PowerPivot, but I'm not sure if all of it can. One of the things I'm doing in VBA is creating a 'future' sales data set, i.e. taking the last 12 months data, duplicating it but adding a year onto the date and changing the "Value Sales" etc to "Value Sales YA". This can probably done with PowerQuery but I'm a basic user and can achieve this in VBA which I'm far more comfortable with.

Microstrategy - Unfortunately the data service where the data is pulled from isn't in house and I can't see any options to run advanced reporting. I can see two reporting options, adhoc and trended reports. Adhoc is hard set to only showing the absolute latest data by MAT, QTR & Latest Period. Trended splits it out by month.

You're probably wondering, well if you can get the MAT & QTR reports why not just use those? Well I could, but that would require importing two reports - it's a potential option. The reason I have monthly data is the report serves two functions (i.e. I see it as efficient to have one data source that just needs rolling up)
1) Monthly pivot chart where users can select via slicers to see performance YoY, either looking at total sales or at certain segments and looking at different facts (Packs, volume and value). This is already up and running and working very well

2) Is a roll up of market shares for the total market and retailer level. We only ever look at this MAT and Latest QTR and would like to look at our share at different levels. I.e. our share of volume, value or TDPs on either the MAT or 12W
 
...
But if I create columns then I can't use a slicer as slicers show choices within one column/field.

Well you can it would be MAT & "Not MAT" in the same field. "Not MAT" would be meaningless but it it would give them MAT to click on.
I imagine this might be a step of abstraction too far for your audience.
Fundamentally MAT & QTR make no sense together (and can't be) in the same column, and can't be in the same slider. As they overlap. This is why your query isn't working.
I don't see any reason to normalize date into another table. Its just duplication and complication.

...
Whilst I see what you're saying about two slicers, a slicer for each month would be cumbersome because you will need to select multiple items in two slicers. For example, the MAT based on January 2021 would require selecting 2020 & 2021, with month containing a multiple selection of Feb through to January.

All I can say is trust me when I know the limitations of my colleague's ability to translate MAT into selecting the right slicers.

I hear yah. Well if thats your audience I'd create two reports. KISS (https://en.wikipedia.org/wiki/KISS_principle) etc.
I'm not into complication. I generally come in take a process that someone, or a few people have layered up that takes hours if not days to run, simplify it, and automate it.
But you have to do it in way you can support going forward.
 
I'm not into complication. I generally come in take a process that someone, or a few people have layered up that takes hours if not days to run, simplify it, and automate it.
But you have to do it in way you can support going forward.

This is so true. I hate having to revisit someone elses system which is partially automated but lots of weird manual steps and if something has broken, continuing it.

The latter part is key if your doing it at work, the time invested in getting it fully automated and robust will mean very little maintenance by you for months/years to come.
 
I have potentially found a solution but haven't had time this weekend to put it into practice. Not entirely 100% on the understanding of the solution, it was posted online somewhere as pseudo code but I think I roughly understand what it's trying to do.
 
Thanks for the post Dude. You sound like a great source of information and experience!

Have to ask on this part, when I had to get stuck into BI, you could download your own personal copy of BI and use it. However when it comes to sharing the data and the like, for business they need a place to run it from like a Sharepoint or Azure cloud space?

This made me u-turn on a project once as I could not get the organisation to provide a place to get the users to view the data. I reverted back to the Power Views and query/pivot but found this lacked a lot of features (it was on 2016 which may have changed now with 2019/365 versions). I like to brush up on this information as I also tend to be half consultant, half provisioner lately in my work and knowing the limitations and better options is certainly useful to stop wasting time and resources.

Yeah, Power BI desktop is standalone and you can share to people online in the cloud or on prem as an interactive report, but it costs a monthly license for each report producer and consumer.
This assumes that you have an azure active directory instance for license provisioning and to allow sign in to the office portal.

There are ways of hosting so non license owners can see (like embedding on an intranet) but this is more premium capacity licensing and sharepoint/office services.
If you don't require to limit data access for security or governance and only have a few consumers that aren't that dependant, then distributing the pbit files is good, it's the template file for the report without data.
Then they can direct import or refer to the database.

What tends to be a good middle ground is to have a Power BI report with sections and pages, each designed for breakdowns filtered in the way a consumer would like to see already.
Then you just refresh the data and export a PDF each report period. They can't interact with a PDF but they get what they want to see.
I've worked on this approach a few times, just remember for pdf exports to set the page size to the same on every one before designing the visuals or enjoy re-scaling all the fonts.

I found people preferred this method as some aren't confident in how they slice and filter the data, but your export breakdown is validated to be correct in a redistributable format that should look very aesthetic.
 
Last edited:
Well bad news is the solution I found doesn't work:
https://www.mrexcel.com/board/threads/create-period-slicer-in-powerpivot.936104/

I get the principle (I think), but their post is a little confusing as they run a min function on the timeframe ID but that measure (SelectedOption) isn't used in their solution. Plus using a slicer to control that unconnected table doesn't change the SelectedOption measure, it always equates to 1 if you have 1 to 3 as your timeframe ID (1 for MAT, 2 for QTR and 3 for Latest Month).

On the plus side I've built some incredible tools with PowerPivot this last week so I've learned a lot already. Might just admit defeat on this one and use a time line filter.
 
but it costs a monthly license for each report producer and consumer.
This assumes that you have an azure active directory instance for license provisioning and to allow sign in to the office portal.

What tends to be a good middle ground is to have a Power BI report with sections and pages, each designed for breakdowns filtered in the way a consumer would like to see already.
Then you just refresh the data and export a PDF each report period. They can't interact with a PDF but they get what they want to see.

This was the issue and being the only person pushing to get it I gave up as people wanted the data but didnt want to help setup the platform.

When I did park it, I recall it only being able to share via PowerPoint presentation files or something which put me off. Its probably progressed more like you say but I couldn't be arsed fighting for it solo. I think two years on now they departments I work with would be able to implement the licenses or server space so its might be worth me chiselling away as BI views in front of the back end data seem to be pleasant to work on.
 
W
On the plus side I've built some incredible tools with PowerPivot this last week so I've learned a lot already. Might just admit defeat on this one and use a time line filter.

Sad to hear that mate. I kind of knew your likely route to salvation was going to stick to a data then slice that.

Ed - just seen you cracked it well done!
 
Sad to hear that mate. I kind of knew your likely route to salvation was going to stick to a data then slice that:

See my last post, I manged to find a resource online in the end which worked. :)

It's basically taking a disconnect table, returning an ID using a custom measure with a min function on an associated ID number to my timeframe description such as MAT, QTR and Latest Month.

Then a switch statement that changes a custom measure to MAT values if the ID number = 1 etc.
 
That makes no sense to me even after a re-read lol. Probably due to being up since 7am and a long day!

Definitely handy to have a Power Q thread though, its good to bounce off others using it. My trouble is it could be months before I revisit the Apps so I tend to forget a lot of what I learned, its far easier to be continually working on the same software day to day - must be my old grey matter wearing out.
 
Back
Top Bottom