I have been posting a lot from the Data Flow in Power Platform, because I needed to use it internally. So far this tool is extremely interesting, but I am not sure it is 100% clear to me how it all works.
This is why i decided to import 94.000 contacts to 1 account, to see how fast/slow it was. I also wanted to monitor how much this affects the API or any other stuff in the CDS/Dynamcis base. This post is mostly for my own curiosity.
I decided to start with a Dynamics instances that didn’t have any users active, and no data. The following data is gathered from https://admin.powerplatform.microsoft.com.
Excel rows = 94884
Excel file size 4,7 megabytes
Excel File for Sample Data – Source for my list. I did clean it a bit before importing it in, and added a column for Account Number = 1234
Data BEFORE the import
I include a lot of pictures for the “before” situation. I was curious to see what happens after.
As always the first thing we do is setup the Alternate Keys in Dynamics. For account, I was only using account number.
For contact I was using email as the unique identifier
Look at earlier post if you wonder how to upload the file to DataFlow. Integrations
This is what my Excel file looked like. I am using First, Last, Email (id) and Account for lookup.
Set the Account Lookup
The E-Mail is the unique identifier for contact
The work around
Open this post to understand why Customer lookup fails, and how to get around it. I will skip to the results here instead. https://crmkeeper.com/2020/03/10/data-flow-set-customer-lookup/
The Result – “Run Project”
The run itself didn’t show much of timestamps, so i opened an advanced find
So it used 1 hour for 94883 records. Not really sure how we can measure this anymore, because of speed caps hitting a little randomly. I would say this is decent speed when importing.
94883 contacts / 60 minutes / 60 seconds = 26,35 contacts pr second
The storage grew a fair bit….
2.72GB -> 3.44GB = 720 MB = 0,007 MB pr contact
This is the old Address entity.. Anyone even use this anymore???
How about API calls?
The first time i ran this, it was only create. Then i did a second round, and all I got was 13,544 API calls. I am not sure I trust the Analytics yet!
After running the import 2 times, i never got any indication about API usage. There must be something wrong with the statistics here? I did almost 100k upserts, and there is nothing to prove that I have accept for the Creates under performing operations. This alone would make this an unbelievable integration platform for CDS because of the API’s we are saving. I understand that this is probably not the case, and there must be something wrong with the reporting.
That being said. No one has been able to tell me if this cost any money. If this is considered a part of Dynamics/PowerPlatform license i am impressed. A truly hidden GEM.
I really hope they continue to work on this product to make integrations simple! It’s the gives me the power to integrate without knowing code!! #lessCodeMorePower
4 thoughts on “DataFlows – Behind the Scenes”
Hi Thomas, that was interesting. Re the API total – I thought we were limited to 10,000 API calls for one user? Are you saying this model ignores that limit?
I am saying that I don’t understand it. I for sure thought this was api heavy, but the stats don’t show it.
I am wondering if the stats might be wrong, but no idea at the moment.
I am doing a new post with ssis to see how it plays with the same file soon 🙂
Reblogged this on Nishant Rana's Weblog.