Dynamics HR + CDS / Dataverse

This is not the average Dynamics CRM post, but I was challenged by Malin Martnes to see if we could integrate the competency part of of Dynamics HR with Customer Engagement. The reason we wanted to look into the matter was because we thought it would be really easy!!! Turns out we were wrong.. hehe 😂

The first thing we did was figuring out what tables were accessible in the CDS integration. At the time of reading the blog, the number of entities was not a lot. The ones that I needed were not there. The complete list of tables that write directly to CDS can be found here:
https://docs.microsoft.com/en-us/dynamics365/human-resources/hr-developer-entities

A lot of fields available, but just not the ones regarding competency.

What to do?!?!? Docs to the rescue !!

Thanks to a link from DOC’s I learned that Finance and Operations has a link to export data. You can choose whatever dataset you want, and export it to file. This has actually nothing to do with HR, but is a feature from Finance and Operations.

https://docs.microsoft.com/en-us/dynamics365/human-resources/hr-admin-integration-recurring-data-export

So I followed the blog but I used Flow instead of Logic Apps.

First start off by creating an export function in HR. Whenever this is called, it will answer your call with a file.

The flow looks like the logic app in structure, but I do a little extra in the end to extract the .zip file to excel.

The first hard part was actually finding the URL for HR. If you ask me it’s quite hidden. I had to have some help from Malin to figure out what my URL to the environment was.

The “Export Workers” has to be identical to the name of the export package. Then you have to decide what the name of the file I wanted to export was.

{
"definitionGroupId":"Export Workers",
"packageName":"talent_package.zip",
"executionId":"",
"reExecute":false,
"legalEntityId":"USMF"
}

The delay is there because the service can sometimes use a little time with the response. Just let it work:)

This part of the flow was just to see if .zip file had been completed. When it was complete, we could get the actual package.

At this point I had absolutely no idea what I was doing, but it was working. The body of the HTTP GET function returned a .zip file. This I could create directly in a OneDrive connector.

The last step was then to extract the file from the .zip, and voila. I now had an automated Excel export from Dynamics HR.

For the last part I could simply connect a DataFlow for importing to Dynamics:)

Was there even a point to this?

Well.. Yes, and no.. The positive thing about this was learning that Finance and Operations has an export function that I could use for extremely simple integrations. This I might be able to use at some later timer.

Later Malin told me that they had released the Virtual entities for Dynamics HR and CDS. You can read more about these here:
https://appsource.microsoft.com/en-us/product/dynamics-365/mscrm.d365hr_virtual_entity?tab=Overview
and
https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4HfPD

I don’t know enough about the virtual entities yet, but I will have a go at that soon to see how it works with using the HR data within Dynamics.

The end goal is still to match competencies from HR to Field Service Dynamics 👍

Dataverse for teams DV4T – Exploring Data Migration

I will keep calling it CDS Lite in the smallest hope that this one day might get the name CDS Lite. It just makes perfect sense to give it a name like this. Allright, enough about the hope for a better naming future…

I know the system is only in preview and many things will change in the next months, but I thought I would give it a go at importing data from CDS to CDS Lite. I had no idea if this was possible, but I was curious to see how it would react.

At the moment there is no connecting to the database through normal tools I have had before

XRM Toolbox
SSIS – KingsWaySoft
Excel Import

Every time I try to connect I get the response saying that I have to use an Oauth method to connect.

PS: Notice how Microsoft has put “LITE” in the error message? It’s a small hope, but it’s a hope 😂

So the only way I found out how to insert data in the Preview was via flow

Flow to the rescue

I am pretty sure that the current doc recommends using Flow for data to CDS Lite (Oakdale), but I am sure this will change in the future somehow. How else are you going to fill 2 million records (estimates from docs on the 2 gig DB size). Flow is not intended for data migration, but I had to use the tools I had available at the time 🙂

First I imported 5694 accounts to my CDS database (top row headings)

The CDS Lite (Oakdale) was empty for accounts, and only included the following columns.

The flow was a little tricky. I actually had to create the flow from the CDS Lite (Oakdale) environment. It did not work otherwise. Current environment connector was fine for writing to CDS – Lite (Oakdale).

The flow will only do 512 records if you don’t open up the paging. Go to settings of the connector and increase the threshold.

I started the flow and patiently waited.

27 minutes later the accounts had been migrated with a warning. I could only find 1 error in the migration, so I consider that a success. All of my accounts were now over to the CDS Lite (Oakdale).

What next

This is all just a beginning of a longer journey for the CDS Lite (Oakdale). Everything is in preview, so there are many things that will change once this is released, and a few years from now. I just wanted to see how to get data in to the system. The reason why might be more clear in a later post where I describe what I see in a future customer scenario for CDS and CDS Lite.

DataFlows – Behind the Scenes

I have been posting a lot from the Data Flow in Power Platform, because I needed to use it internally. So far this tool is extremely interesting, but I am not sure it is 100% clear to me how it all works.

This is why i decided to import 94.000 contacts to 1 account, to see how fast/slow it was. I also wanted to monitor how much this affects the API or any other stuff in the CDS/Dynamcis base. This post is mostly for my own curiosity.

I decided to start with a Dynamics instances that didn’t have any users active, and no data. The following data is gathered from https://admin.powerplatform.microsoft.com.

Data:
Excel rows = 94884
Excel file size 4,7 megabytes

Excel File for Sample Data – Source for my list. I did clean it a bit before importing it in, and added a column for Account Number = 1234

Data BEFORE the import

I include a lot of pictures for the “before” situation. I was curious to see what happens after.

Import

As always the first thing we do is setup the Alternate Keys in Dynamics. For account, I was only using account number.

For contact I was using email as the unique identifier

Look at earlier post if you wonder how to upload the file to DataFlow. Integrations

This is what my Excel file looked like. I am using First, Last, Email (id) and Account for lookup.

Set the Account Lookup

The E-Mail is the unique identifier for contact

The work around

Open this post to understand why Customer lookup fails, and how to get around it. I will skip to the results here instead. https://crmkeeper.com/2020/03/10/data-flow-set-customer-lookup/

The Result – “Run Project”

The run itself didn’t show much of timestamps, so i opened an advanced find

First Record

Last Record

So it used 1 hour for 94883 records. Not really sure how we can measure this anymore, because of speed caps hitting a little randomly. I would say this is decent speed when importing.
94883 contacts / 60 minutes / 60 seconds = 26,35 contacts pr second

The storage grew a fair bit….
2.72GB -> 3.44GB = 720 MB = 0,007 MB pr contact

This is the old Address entity.. Anyone even use this anymore???

How about API calls?

The first time i ran this, it was only create. Then i did a second round, and all I got was 13,544 API calls. I am not sure I trust the Analytics yet!

Conclusion

After running the import 2 times, i never got any indication about API usage. There must be something wrong with the statistics here? I did almost 100k upserts, and there is nothing to prove that I have accept for the Creates under performing operations. This alone would make this an unbelievable integration platform for CDS because of the API’s we are saving. I understand that this is probably not the case, and there must be something wrong with the reporting.

That being said. No one has been able to tell me if this cost any money. If this is considered a part of Dynamics/PowerPlatform license i am impressed. A truly hidden GEM.

https://docs.microsoft.com/en-us/power-platform/admin/data-integrator

I really hope they continue to work on this product to make integrations simple! It’s the gives me the power to integrate without knowing code!! #lessCodeMorePower

DataFlows – Set parent lookup

This blog post is step 3 of the Data Flow setup in combination with On Premise Gateway. In the last post in imported all of our ERP accounts to the CDS database. Now I wanted to import all active projects, and connect them to the the Accounts in CDS. There are a few odd things you have to do in the setup to make this work, but I will try to outline this without being to detailed..

First off we need to get familiar with the Alternative Keys in Dynamcis. In Account I created alternative key for AccountNumber. I need to do the same for Project.

The field is also represented on the form. I choose to keep these as “Single Line Of Text”. Just keep it simple, because this doesn’t need to be advanced.

This is important so the import is able to match every time runs. Remember that the Data Flow is doing UPSERTS. It will create a new record if it can’t find the project number. Imports CAN’T connect to GUID’s because my ERP system does not have the CRM GUID.

Make.PowerApps.com

The most important thing you do here, is make sure you have the correct environment!! 😉

If you don’t understand this setup, look at my first blog about the DataGateway

The important pieces of information here are:
* Project Number
* Project Name
*Account Number

This is all i need to create a project in CRM and connect it to the Account

Connecting to Lookup

This is where you need to pay close attention. First you choose the project (or other entity) from a list of existing entities. The Alternate Key will display if you have done this correctly. If nothing is there you have to go back and start again with alternate keys

There are 2 lookups on project. Account and Contact. Account has 2 alternate keys, and Contact has 1 alternate key.

Account:

Contact:

This is why you will see 3 fields in the beginning of the mapping
1. ncg_Account.AccountNumber – Account looup identifier
2. ncg_Account.ncg_POAccountNumber – Accound lookup identifier
3. ncg_CustomerContact.EMailAddress1 – Contact lookup identifier

I am using the AccountNumber for a different purpose, so let’s just forget this one for now. I am using the Account Number from ERP to the ERP field on the Account entity to create a match.

Schedule

After you have set a schedule, you should see the results in CRM:) In a later post I will tell you why I am connecting the project records to dynamics. A quick hint, Forms PRO and NPS!

CDS / Dataverse List Records Filter Query using Flow

This week I needed to use the List Records function, and I realized that I had no idea how to use the filters. Thank you Jonas Rapp for creating the FetchXML Builder!! The function “Flow List parameters” saved my day:)

Simple filters

Let’s begin with the simple filters where I get a contact with the last name of Sandsør

Test your search result with the Execute button so see that anything is actually returned. Then open the Flow List Parameters

The tool converts the Fetch XML, and magically gives the correct filter to add in our FLOW query. It can’t get much simpler than that!!

Lookups

Lookups act a little bit different with the syntax, as lookups always to. This got me quite confused before finding this tool, because I was not getting match to my result.

I am searching for contacts with a given GUID. In my case I didn’t know what the GUID was, so I randomly generated a GUID for the formula. In FLOW I substituted the GUID part of query with a dynamic variable.

Filter Query with lookups, you need to add “_” as seen above. When working with lookup you won’t get at match without the “_LookupField_Value”.

Filter linked entity

The last filter is a little more complex, and might not get used due to some limitations of Odata (Must match on unique ID for related).

In this scenario I wanted to locate all contacts with last name “Sandsør” where the regarding accountID = GUID.

Choose the main entity on top, and add “link-entity”

Make sure you have the correct relationship here. Some Lookups support more than one entity, and therefore you make sure you have the correct one.

Again we find the magic with the “Flow List Parameters“.

In this scenario we also get Expand Query result that we need to copy/paste.

Apply to Each

Once you have figured out what filter to use, you can select the “Apply to Each” function, and add custom logic in here.

Flow – Create, Update or Delete trigger error

At the time of writing this post Microsoft Flow will throw an error when you create this flow with the trigger Create, Update or Delete, and the flow is standalone.

Error

This is the following error you will see when you try to create the flow (Create, Update, Delete) without any connection to a solution.

Sollution

All you have to do, is open this in a solution (for the time being).

Dynamics 365 VOC end of life

Dear VOC,
Thank you for services provided, but you have been replaced with a younger and more exciting model.

During a call today Microsoft made it quite clear that you should start the transition to FormsPro. On the questions regarding what we do with existing customers, the answer was pretty clear “you should transition”. The one thing that I did think was a little strange was the fact that Microsoft said they would provide assistance migrating (a tool of course). Usually Microsoft doesn’t provide any form for tool for transitioning, and make the partners do this manually.

The magic

I might sound negative, but honestly I think it is quite amazing. VOC turning into a separate app on top of the CDS only commits that MS is really serious about the PowerPlatform story and APPS. Microsoft chose to store data in the CDS, so the options for this tool are almost endless. And even more, you don’t need Dynamics to deliver the product.

Licensing

So what will happen to the product now that it is no longer a Dynamics product? Good question. It was mentioned that licensing is in the works, so let’s hope that this won’t become another Marketing “scandal”. The only thing I would prepare for is a license that now cost more than what you pay today. You can hope for the P2 license to cover this, but probably no.

No one used VOC besides Dynamics customers with the license included. Being able to configure this product separately, you can use any system with a CDS connector. This will open a marked for the FormsPro that the VOC never had. The demonstration was used with Salesforce and Dynamics.