CRM 2011 Performance: Take advantage of better query performance for your Quickfinds!

Hey everyone, as many of you may know we made some pretty big changes to how CRM retrieves data from SQL server, starting in UR10. Part of this was adding various measures and statistics to change the queries based on the dataset and the users access to records, we also added the ability to “hint” to CRM how it should structure a query based on a property called isQuickFindFields.  The property was created specifically for quickfinds but if you’re like our CRM PFE’s you’ll want as much control as possible for how queries interact with SQL server.  If you’ve been looking for a way, especially in extensibility scenarios, to better control performance via index tuning and smart querying then you’re going to love this!

Now for the comparison: I’ll show you the differences between a quickfind without this optimization (default pre-UR10) and a quickfind with this optimization (post UR10).  You’ll find a super zoomed out clip of the query plans below, and while they’re impossible to read they still illustrate how the query plan is split up differently based on these settings.  The union based query splits up such that it could be filtered in multiple spots and potentially leverage parallelism (big opportunity to put index tuning to work here).  Compare that to the standard (preUR10) query which joins everything together and many times cannot filter the dataset until the end of the plan, this isn’t always the case but no matter what the SQL optimizer will do whatever it can to try and give us a decent query plan given the circumstances.  Post UR10 CRM uses a Common Table Expression along with a Union effectively telling SQL that it can go process all the individual datasets separately (including filtering them) then come back and union the results together – meaning it will filter earlier in the process if it can. 

Please note: this new query is *not* necessarily always more efficient, thus CRM will change the approach for various users and sizes of datasets, but in many cases this query offers us more flexibility when dealing with large datasets and leveraging custom indexes; essentially it’s another tool in our toolkit of options we can use when we’re tuning and optimizing.  Another note, this is enabled by default for quickfind queries today and no other views by default.  Want to know more?  Want to get down to the fine details of what things can be tweaked?  Read more here in the Optimization White Paper.  I want to emphasize, query plans will change depending on your SQL server and all the statistics SQL uses to make informed decisions – these are common cases below. 

Default out of box query prior to UR10:

execsp_executesqlN'select
top 251 "new_labor0".New_WorkCodeId as "new_workcodeid"
, "new_labor0".New_Minutes as "new_minutes"
, "new_labor0".New_AccountId as "new_accountid"
, "new_labor0".New_comments as "new_comments"
, "new_labor0".New_LaborDate as "new_labordate"
, "new_labor0".CreatedOn as "createdon"
, "new_labor0".new_projectid as "new_projectid"
, "new_labor0".crmfargo_LaborCaseId as "crmfargo_laborcaseid"
, "new_labor0".New_laborId as "new_laborid"
, "new_labor0".New_WorkCodeIdName as "new_workcodeidname"
, "new_labor0".New_AccountIdName as "new_accountidname"
, "new_labor0".New_AccountIdYomiName as "new_accountidyominame"
, "new_labor0".new_projectidName as "new_projectidname"
, "new_labor0".crmfargo_LaborCaseIdName as "crmfargo_laborcaseidname"
from New_labor as "new_labor0" (NOLOCK) 
where
(((("new_labor0".New_WorkCodeIdName like @New_WorkCodeIdName0 or "new_labor0".OwnerIdName like @OwnerIdName0 or "new_labor0".crmfargo_LaborCaseIdName like @crmfargo_LaborCaseIdName0 or "new_labor0".New_comments like @New_comments0)))) order by "new_labor0".New_LaborDate desc, "new_labor0".New_laborId asc',N'@New_WorkCodeIdName0 nvarchar(9),@OwnerIdName0 nvarchar(9),@crmfargo_LaborCaseIdName0 nvarchar(9),@New_comments0 nvarchar(9)',@New_WorkCodeIdName0=N'Contract%',@OwnerIdName0=N'Contract%',
@crmfargo_LaborCaseIdName0=N'Contract%',@New_comments0=N'Contract%'

Fig1: Common pre-UR10 query plan:

* notice how the data (thickness of the gray bars) continues to grow towards the upper left, it then all joined together using a hash join to put it all into one large dataset, then it is sorted and filtered.  This illustrates how all the data is combined together then sorted before it can be filtered which can be less efficient, especially with some datasets (in others this is much less of a problem).

With the isQuickFindFields property enable (post UR10) here is the difference in the TSQL query and the plan – also note how there is now an inline count:

execsp_executesqlN'WITH __QuickFind__ as (select top 10001 [New_laborId] from (
SELECT "new_labor0".[New_laborId] as [New_laborId] from [New_laborExtensionBase] as "new_labor0" where ("new_labor0".New_comments like @New_comments0) UNION
SELECT "new_labor0".[New_laborId] as [New_laborId] from [New_laborBase] as "new_labor0" where "new_labor0".OwnerId in (select [OwnerId] from [OwnerBase] as "new_labor0" where "new_labor0".Name like @OwnerIdName0) UNION
SELECT "new_labor0".[New_laborId] as [New_laborId] from [New_laborExtensionBase] as "new_labor0" where "new_labor0".New_WorkCodeId in (select [New_workcodeId] from [New_workcodeExtensionBase] as "new_labor0" where "new_labor0".New_name like @New_WorkCodeIdName0) UNION
SELECT "new_labor0".[New_laborId] as [New_laborId] from [New_laborExtensionBase] as "new_labor0" where "new_labor0".crmfargo_LaborCaseId in (select [crmfargo_laborcaseId] from [crmfargo_laborcaseExtensionBase] as "new_labor0" where "new_labor0".crmfargo_name like @crmfargo_LaborCaseIdName0)) as [__QuickFindInternal__])select
top 251 "new_labor0".New_comments as "new_comments"
, "new_labor0".New_Minutes as "new_minutes"
, "new_labor0".New_LaborDate as "new_labordate"
, "new_labor0".New_AccountId as "new_accountid"
, "new_labor0".OwnerId as "ownerid"
, "new_labor0".crmfargo_LaborCaseId as "crmfargo_laborcaseid"
, "new_labor0".New_laborId as "new_laborid"
, "new_labor0".New_AccountIdName as "new_accountidname"
, "new_labor0".New_AccountIdYomiName as "new_accountidyominame"
, "new_labor0".OwnerIdName as "owneridname"
, "new_labor0".OwnerIdYomiName as "owneridyominame"
, "new_labor0".OwnerIdType as "owneridtype"
, "new_labor0".crmfargo_LaborCaseIdName as "crmfargo_laborcaseidname"
, case when (select COUNT(*) from [__QuickFind__]) = 10001 then 1 else 0 end as
[__QuickFindLimitValue__]
from New_labor as "new_labor0" (NOLOCK) 
where
[new_labor0].[New_laborId] in (select [New_laborId] from [__QuickFind__]) and (((("new_labor0".statecode = @statecode0)))) order by
"new_labor0".New_LaborDate desc
, "new_labor0".New_laborId asc',N'@statecode0 int,@New_WorkCodeIdName0 nvarchar(9),@OwnerIdName0 nvarchar(9),@crmfargo_LaborCaseIdName0 nvarchar(9),@New_comments0 nvarchar(9)',@statecode0=0,@New_WorkCodeIdName0=N'Contract%',@OwnerIdName0=N'Contract%',
@crmfargo_LaborCaseIdName0=N'Contract%',@New_comments0=N'Contract%'

Fig2: Common post UR10 query plan:

At face value this looks much more complex, however if you dig in you’ll find the plan to be more efficient.  In this case each portion of the plan can be processed and filtered independently from the other, which can have dramatic reductions in the amount of data that needs to be processed resulting in much less expensive queries.

Here are some questions I have received around this change:

Q: This is AWESOME – how do I get this? 
A: Great news – everyone can get it or already has it , install the latest rollup and make sure EnableRetrieveMultipleOptimization is set to default. If you’re already on UR10 or higher and have the feature set to zero (or not set at all) you will then have this feature enabled!

Q: What’s an “enable retrieve” … – whatever you said?
A: It’s highlighted in the Optimizing CRM white paper and can be used to control CRM’s query behaviors.  It can be set in the registry for onPrem customers or via your orgDbOrgSettings if you’re onPrem or Online – I personally prefer the orgDbOrgSettings because they aren’t server-specific and don’t require any registry editing. Also, be aware there are a few other settings documented in that white paper that can give you even more control over the query behavior – but most of the time the defaults are fine.

Q: What are OrgDbOrg settings?  
A: OrgDbOrgSettings are available on a per-organization basis and offer you greater flexibility and control of your CRM organization – no matter if you have Online, onPremise, or partner hosted – you all have orgDbOrgSettings for every CRM Organization.  If you are unfamiliar with these, there is a command line tool you can use to change them – however *hint* there may be a, new, much easier way to edit these settings in the near future.  If I were you, I would keep an eye on your favorite CRM blog for more information in the near future! If you want to be kept up to date subscribe to the blog and it will be sure to email you when we post something. 

Q:  I’m a developer or admin and need to extend CRM, I want to use this to be used in my custom queries, where can I use the new isQuickFindFields property? 
A:More great news!  Custom fetch reports can use this fetchXml isQuickFindFields property setting, it can also be used for any fetch statements you’re executing from code. Also, if you have a QueryExpression (in managed code, plugins, etc) you can set the “isQuickFindFields” property on the FilterExpression – be sure to use the latest SDK libraries for managed code scenarios as these will expose the new property.  This query change is not used when querying via the REST endpoint. 

Thanks everyone!

Sean