David Taylors Business Intelligence Blog

On August 5, 2009, the SAP BusinessObjects Innovation Center announced a prototype for moving Sets Technology directly into the Semantic Layer.

Potential Future of Sets

Since the EOL announcement of BusinessObjects Performance Manager and two of the three analytic engines, Process Analysis and Predictive Analysis, there has been an ongoing question on the future of Set Analysis.  I have long been an advocate of the incredible power and importants of sets to help customers get a better understand of how data is changing under the covers.  If you have 100 more customers, that’s great… but how many did you lose and how many did you gain?  You may have lost 50 and actually added 150.  What about my “Gold” Customers?  How many new Gold Customers do I have?  Where did they go?  Did they become Silver members?

We have many customers who have successfully implemented Set Analysis; however the solution had not had significant R&D investment for some time and was tied to the legacy metrics engine of Performance Manager.  How could this technology be integrated into the new XI 3.X platform?

Sets and the Semantic Layer – I’m in Heaven

The Innovation Center has delivered a prototype which show Sets integrated directly into the Semantic Layer.  This prototype is a plug-in to the universe that allows you to create these “sets” , static or dynamic – all in the query panel.  Semantic layer sets lets customers leverage the complexity of  sets natively within that environment.  This is a huge step forward.

Check out the 15 minute video on the SDN website

You can see that this implementation can support all the classic calendar sets which will support of joiners, leavers, stayers, which are at the core of Set Analysis.  In this labs implementation there is support for both Visual Data and Visual Sets; however I hope this is expanded in the future to support Freehand SQL and an imported list.

I think this is a fantastic new innovation and needs to be added to the core product ASAP.  If you think so too, let your voice be heard:  http://sets.uservoice.com

I’ll keep you up-to-date with additional innovations around sets and may provide some additional posts on this topic if there’s interest.  Let me know.

«Good BI»

So how we do do this exactly? How can I use BusinessObjects to create custom groups on-the-fly? How can I get beyond my requirements for multi-pass SQL and speed up my reports? The answer is Sets.

Set Data

All the information regarding sets is located is the tables called sets_*. The most important tables are:

• set_set – contains the set name, description, etc.
• set_set_detail – contains all the set member information including member key, join date, leave date.

Here is a universe I created that you can download and use with the sample database provided in my earlier blog post. We need to add the set_set and set_set_detail tables to our WebIntelligence Universe so that the set member information they contain can be used within our query.

Here you can see that we created a join to the DW_CRM_CONTACT table. We also created a self-join on the set_set_detail table. The self join make sure that we reference the edate value of 12/31/2999, which is the default edate for current members of any set. In the case of a dynamic set, If a member had left the set then the edate would not be equal to 12/31/2999.

In addition to this, we also joined the set_set table to the set_set_detail table using the setversion_id key. (It may appear that the setversion_id is always the same as the set_id, but it is not so only join the tables via the setversion_id)

Here is a look at the Custom Product Groups

BEST PRACTICE: Whenever you may need to alias a table. If you alias a table, then all references to that table should be aliased. You can see that because I needed to reference the set_set_detail and set_set tables twice, once for products and once for contacts, I aliased both.

User Prompts

After joining the tables, we needed to create the filters for our custom groupings to make it easy for users to select which groups they would like to use on the report.

I specifically defined the filters a multi, which will allow the user to select more than one filter. This is important because it provides even more power for your end-users. Now they can do direct comparisons of one group to another, even if a member (e.g. contact) exists in more than one group. The only downside is that it means that a member can be counted twice, this is why I recommended that when you define your sets you should use naming conventions that identify the sets as being mutually exclusive.

Here is what the prompt will look like in action:

In my universe I did not use cascading prompts to first limit the sets by folder, but that is certainly an option since folder information is located in the set_project table.

Getting Results

After configurating the universe and setting up the prompts this is an example of the results you will see. I have run a report and selected multiple custom groups which are mutually exclusive. These groups on based on the current periods sales amounts:

Here you can see the report is broken down by my custom groups which were defined and processed by the Sets Engine.

Summary

I’ve really enjoyed taking you on a quick tour of Set Analysis and showing you some of the ways that you might get more value out of your existing data assets. Sets offers so many solutions around data analysis:

• Custom Groupings – which allows the organization to more nimble as changes such as mgmt structures, mergers and acquisitions take place.
• Classic Segmentation – Joiners, Leavers, Stayers (which in turn are perfect for analysis by a predictive tool so that you can see why are people joining or leaving? What are the influencers?)
• Advanced Analysis – “Give me all quad-pack customers who have never purchased phone accessories from a company owned retail store.”

The ability to divide customers up into categories of: highly profitable down through :unprofitable, and using that to drive the business because the bottom line of any business is the customer and your organization’s ability to make a profit.

Although dashboards are currently in vogue and everyone wants one, I often wonder how organizations are analyzing the information beyond the dashboards. I have nothing against dashboards, but I would encourage you to ask yourself this. Once we see that sales are falling, what am we going to do? How am we going to do root cause analysis? Were all customers affected? What about products? Segmentation might well hold the answer to these yet unsolved mysteries.

I recently had a customer who was in the process of migrating from a legacy version of Microstrategy to BusinessObjects. They had been able to migrate over most of their core reports, but there were several reports that required a number of custom groups.

Microstrategy’s approach was to allow the customer to define any number of groups and filters from with the Microstrategy Desktop tool. These global filters and group definitions can then be leveraged by the report editor for building the report. Microstrategy’s SQL generation engine uses multi-pass SQL when creating the reports. Multi-pass SQL means that a single SQL statement cannot deliver back the correct results, so the report data is written into temporary tables. These temporary tables are then combined with additional queries to obtain the final results.

In the case of BusinessObjects, it wasn’t going to be possible to take the same approach. BusinessObjects does not generate multi-pass SQL. Instead, BusinessObjects focuses on creating extremely complex SQL in a single SQL statement. In most cases, this works just as well, but this was an exception. This sales report used a combination of multiple filters at multiple levels in the heirarchy. The Heirarchy was also defined within the custom groupings and not within the database.

The right answer is: Modify the Data Warehouse and ETL processes to make the underlying data better fit your reporting requirements. You see, the problem was that these custom groups should be defined within the data warehouse natively. This would allow for simplified SQL that can be run in a single pass. Translation? Reports the use custom groups will run a heck of a lot faster.

In my situation, the customer wasn’t going to be able to make these changes, but still wanted to migrate the report to BusinessObjects.

The best answer is: Use Sets. Sets gives you the ability to generate those custom grouping within the data warehouse. Unlike Microstrategy which will run the same multi-pass SQL over and over again, sets will allow us to “cache” the previous passes using sets and run single-pass SQL. We can then tie the set tables into the data warehouse and add the custom groupings to our universe.

The advantage of sets over ETL is that it can be much more dynamic. Sets can allow an analyst to create a new set in a matter of minutes and then see a report broken down by the new custom group.

In my case it was just what the doctor ordered. By introducing sets and adding the set tables to the BusinessObjects semantic layer, we now have access to the sets, which in this case, the customer called custom groups. It was perfect.

Now, not only will the customer be able to migrated this complex report from Microstrategy to BusinessObjects, but they will also be able to run the report much faster than ever before because the report will be generated through a single-pass process.

Tomorrow, we’ll take a look at how to set this up using our simplified data.

A less common way to create sets is to create sets is through free-hand SQL. Normally you will want to leverage to power of the set engine to design the SQL for you to return the correct results, but in rare cases you might find it helpful to create your own SQL Statements.

Free Hand SQL

The only item that needs to exist in the SELECT clause is ID. If an ID is specified and the SQL is valid, then the records will be imported into the set tables, if not, you will not be able to chose “Parse” to validate your SQL.

For our example, we can create a set called “All Customers” and use the following FreeHand SQL.

select CONTACT_ID as ID from dbo.DW_CRM_CONTACT

The SQL may be as complex as you require. As long as the SQL is valid, you should be able to insert it here and use it with Set Analysis.

I am not aware of a limit for the Free Hand SQL.

Import

Set Analysis also provides the ability to import individual records into the set tables. This is valuable if users have exported information into Excel and manually developed a list of records they wish to add to a set.

You can either store the text file in the BusinessObjects Repository or on the physical filesystem. Unfortunately the schema definition is pretty limited, so I recommend you use one of the sample files below as a guide. As long as the file contains at least 1 column and the key is the FIRST column, the import will work fine.

Click on the Import button and pick your file. Do not type in the name manually. If necessary, you may need to define a file format. I recommend that you use comma delimited files with a double-quote text qualifier (”).

Here is a sample import file of DM Contacts.

NOTE: In order to use the Upload feature, a c:\tmp directory must be defined on your system.

NOTE: If you store the file on a filesystem, you may need to make sure that the Sets Query Service (AAQueryMgr.exe) has rights to access the file across the network. By default all BusinessObjects services run as LocalSystem which means by default they will not have access to UNC file paths
.

I’ve decided it’s probably a good idea to break down these tutorials a bit. Although I love to share all this information, it’s probably getting a little long for a single post.

Visual Sets

When creating sets in Set Analysis, Visual Sets are extemely helpful. Visual Sets allow you to quick look at the union, intersection, exclusion of multiple sets simultaneously. Let’s say for example I want to see all Gold Customers who are also High Tech Customers but have NOT purchased products from our computer department. Let’s assume I already have the following three sets:

• Gold Customers
• High Tech Customers
• All Products from the Computer Department

Let’s create a new static Set using DM Contacts, called Gold Customer – No Computer. Next we will add a Visual Set. We will chose 3 sets. One set for each of the three Collections.

• Collection 1(blue) – add Gold Customers (All Gold Customers)
• Collection 2 (red) – add High Tech (All High Tech Customers)
• Collection 3 (yellow) – add Computer (All Products in the Computer Department)

In our case we want all Gold Customers who have NOT purchase Products in the Computer department, therefore we do not want to select anything in the yellow circle.

I can click on Counts at any time and the system will show many how many records are in the selected set.

From the selected venn diagram regions, I can see from the displayed count that there are 78 customers who fall into this category. Click Next>>, specify the description and OK to save the set.

Remember, just like before we must build the set before it will contain any objects. Once you build the set you will see there are 78 records retrieved.

Isn’t that easy? How cool is that?

NOTE: Remember how we specified Converters in Set Analyzer Architect? This converter allows us to chose information from DM Products, but have it converted products to contacts. In our example, it converted “products in the computer department”, to contacts who have purchased products from the computer department.

w that we’ve spent some time with sets, it’s time to begin to see the technology in action. Business Objects has made sets easy to build and maintain. There is virtually no limit to the ways you can now choose to look at your customers and look at how those customers are contributing to revenue, profit and margins.

Using Sets to Maximize Revenue

Click HERE is a quick online demo that we’ve built to explain how some of our telecommunications customers use sets to maximize revenue, cross sell and up sell opportunities.

Creating Sets

The beauty of sets is the fact that you don’t have to involve IT in making changes to the underlying data structures to segmentation. With the set tables that come with Set Analysis, end users can create their own custom collections of products, customers, etc. BusinessObjects users will then be able to use those sets together with the BusinessObjects Suite to provide in depth reporting of key metrics and kpis with respect to these custom collections of customers. Whether you want to track standardized silver, gold and platinum customers or time, or simply do some adhoc customer analysis, sets gives you what you need.

In XI 3.0 you can create sets using our Performance Manager dashboard interface. A set can contain data for only one subject. The subject is normally identified as a primary key in a dimension table. In the case of the Customer subject, the Customer Id key will be the key we are tracking.

Create Calendar

The single remaining configuration component that still remains is to create a calendar. Calendars are used by BusinessObjects to allow administrators to automate task such as refreshing sets and to allow us to track the changes of sets over time. We need to tell BusinessObjects how often track the changes in our sets.

Based on continuing our demo from the my previous post, lets create a calendar monthly. To create calendar, navigate to the Dashboard & Analytics Setup. Next, choose the Time Config > Calendar submenu. Here is where you can see a list of all available calendars. Let’s choose New Calendar

Based on the same data provided in my previous post, we will create a calendar:

1. Name: DM_Monthly
2. Interval: Monthly
3. Format: First Day of Period, 9/21/04
4. Range: Jul-00 to Mar-03

After you have created the calendar the screen should look like this:

NOTE: Once a calendar has been created, you cannot change values such as interval.

Now that the setup is complete, we can move to Dashboards & Analytics. Chose the Set Analysis > Sets submenu.

The first thing we need to do is create a folder so that we can categorize our sets. Choose New Folder and let’s name our folder: Customer Types.

Let’s also create a second folder called: Customer Segments.

Now let’s create our first set.

Static Sets

The first set we will create will be a static set. A static set is a collection of objects that either:

• Doesn’t change over time or
• if it changes, we are not concerned about tracking those changes.

Some examples of static customer sets are:

1. Customers Living in Montreal
2. Homeowners with more than one child
3. Key accounts younger than 30 years old
4. Customers in the UK with sales over 10,000 in 2001

We will be able to create a set of customers using any of the customer table reference that we setup previously. Let’s create a set of all High Tech GenX Customers, that is: All Customers with Internet and in the 30-40, 40 – 50 Age Bands.

First we need to name our static set and set some attributes for our set. Our set will be called:

1. High Tech GenX
2. Folder: Customer Types
3. Description: This set will contain a list of all customers who are considered Highly Technical, Generation Xers.
4. Scope: Public
5. Set Type: Static

Click Apply. Now we will specify the content of the set. In our case, we will use Visual Data.

NOTE: The interface can be a little tricky. You will see Add/Copy/Remove buttons next to Sets and also find Add/Edit/Remove/Empty button within the Set Building Sets section. The buttons on the left operate on the entire set itself. The buttons on the right only affect the selected building step.

There are 5 was to add information into a set. They are:

• Visual Data – Allows you to look at all the fields associated with your subject (e.g. customers) and visually see how many records will be included in the set. (Most Popular)
• Visual Sets – Allow you to reuse existing sets to create new sets. For example you can visually see the intersection between High Tech GenX Customers and Gold Customers for example.
• Free Hand SQL – Allows the users to specify their own SQL against the database. (Not Recommended)
• Event Analysis – Allows you to create new sets based on a records movement within a dynamic set for a specific time period. For example, all customers who were leavers between Oct-02 and Dec-02.
• Import – Allows you to manually import records (e.g. contact_id values) from a csv or txt file.

We will chose Add > Visual Data and be taken to the window, where we can select the attributes of our set members.

We will chose:

• Has Internet – Y
• Age Range – 30-40 and 40-50

We will select the attribute on the left and choose the arrow to move the attribute to the right. (If prompted, chose Exact Counts). The system will tell you that 1135 contacts have Internet. Choose the Y checkbox.

Next select the Age Band attribute on the left and expand the section. You will see that of those with Internet, 321 are aged 40-50 and 302 are aged 30-40. Select both of these checkboxes. This means that when we build the set, we will expect a total of 623 members.

Now choose Next and fill in the Description: Internet and Aged 30-50

Now we can choose OK and we can see the resulting set.

We could continue to add additional steps, which would allow us to modify this set further (allowing us to add or subtract members), but our example will only have one step. Now we need to build the set.

Building Sets

Building Sets is the process of the Set Analysis engine using the definition of the set to generate the appropriate SQL statement(s) that brings back the correct records. It will then insert those records into our set tables for future reference.

In the screenshot above, you can see the cursor next to the Build… icon. Click build now and watch as the set engine identifies 623 members for the set. Until you build the set, your set will have zero members. You an always see how many members a step and the last date/time the set was rebuilt.

If at any point you edit the building sets of a static set, the set will be automatically emptied (all set members will be deleted) and you will need to rebuild the set.

Dynamic Sets

The process of creating dynamic sets is very similar to that of static sets, except the set will be processed for specific time periods. Let’s create a dynamic set of Gold Customers. Gold Customers are customers who have made volume purchases and purchased more than $3000 of products from us in each period. We will again use the Visual Data to build our new set.

First we need to name our static set and set some attributes for our set. Our set will be called:

1. 1 – Gold Customers
2. Folder: Customer Segments
3. Description: Our Worldwide Gold Customers
4. Scope: Public
5. Set Type:Dynamic
6. Calendar: DM_Monthly
7. Membership History: Temporal (allows entire history of set membership to be kept)
8. Allow Auto-Fill: Yes

Click Apply. Now we need to build our first step. Let’s choose Add > Visual Data.

First, when specifying dynamic, temporal sets, it is important to specify the time element as the first component of the set detail. We want to make sure we are only including customers who have made a purchase is the current period. Because the sales date is part of the Orders Subject, we need to choose DM Orders.

Use the arrow to move the Start Date to the right hand side. You will be prompted to specify the Added Field Condition. We will need to specify, Start Date, between, with the values of Relative, Fixed, start of processing period and Relative, Fixed, end of processing period. The Yes/No condition should be selected.

In our case we will see that 18107 records are No. By default No may be selected. Make sure and change it to Yes.

TIP: Usually you will want Yes/No conditions to be Yes, so under Options (above the first Yes/No Condition line), you can chose the setting: Default Yes/No conditions to Yes.

Next, let’s look for Contacts who have purchase at least 12 items from us in a single monthly period. So chose Sum of Volume. You need to be careful here. Notice how we chose Sum of Volume. This is critically important, otherwise it would only choose customers with sales of one transaction with a Volume of 12. That’s not what we want. We want to look at total volume for the entire period.

Finally we want Contacts who have purchased more than $3000 of items in the same period. Let’s chose Sum of Actual Sales greater than 3000.

Now chose Next>> and enter a description: High Volume, High $$ and click OK.

After creating the set, BusinessObjects gives you the option to Autofill the set. The Autofill process uses the set engine to populate the historical values for the given set to a predetermined number of periods. In our case, we will autofill all 17 monthly periods from Jun-00 to Sep-03.

Click on AutoFill/Rollback…, Complete (radio button) and OK.

Now the request to process the set historically will be sent to the Set Analysis engine and after it processes all 32 periods, you can click on the Membership view and see the an historical view of the set.

Here you can see each period and the number of joiners, members and leavers.

Create two more sets for Silver and Bronze customers with volume purchases. I recommend Silver as Volume: 12 or more, Sales: 1000 – 3000. I recommend Bronze as Volume: 12 or more, Sales: <1000.

NOTE: If you want to get more members for each period, you can drop the volume requirements and only look at Amount of Sales.

Tips For Organizing Sets

It is important to note that you cannot create subfolders, so I recommend you create development standards when it comes to sets. Because all folders and sets are organized in alphabetical order, I recommend that you use the colon “:” to differentiate folder depth.

For example:

Level1
Level1:Level1a
Level2
Level2:Level2a
Level2:Level2b

In the same way, use colons, numbers and names to organize your sets (1, 2, 3 or A, B, C will help create order.) For example, if I am going to create three sets which are mutually exclusive, such as Gold, Silver, Bronze, I would name them:1-Gold, 2-Silver, 3-Bronze. Now they will appear in the correct order. If however I am going to put more than 1 set in the folder, I might called it ProjA:1 – Gold, ProjA:2 – Silver and ProjA:3 – Bronze. That way I can call the next ones ProjB, etc (or whatever project name made sense in my situation).

Try to keep set names to less than 25 characters.

Consider using other symbols such as asterisk(*), plus(+), etc at the end of the set name to indicate addition important information (such as the fact that this set depends on another set).

If you don’t organize your sets, they will become cluttered. Another advantage of these types of naming conventions is that all this set information is stored in a relational database, so we can create BusinessObjects reports which can help us keep our sets organized. We can parse out colons an other special characters which may have special meaning to the organization.

Read more…

I have found that setting up the Metadata around Sets was one of the most confusing things about learning sets. The problem was two-fold. The interface of Set Architect is old and not terribly intuitive. It is little more than a database table browser with a few rules built in.

I typically learn by example, so I put together what I felt was the best combination of screenshots to show you the core of what you will likely run into when you need to configure Set Architect.

To download the sample database that we used for this example, please click here.

You may find trouble reading through the Set Architect manual before install the Set Analyzer client because the document isn’t available on our website. The document is contained within the installer program but I thought it might be a good idea to make it available here in case you need it.

• Set Analyzer Administrator Guide – Administration and Setup Guide, which primarily uses Set Architect (required for using Sets)
• Set Analyzer User Guide – Legacy documentation on how to use the Set Analyzer client tool. This tool was made obsolete because the primary interface for sets has been integrated into the Business Objects Performance Manager dashboard interface.

Metadata Configuration within SetArchitect

There are four core components of the metadata that need to be configured from within Set Architect. They are the:

• Tables/Objects – the database tables and the field definitions we will use when generating SQL to create our sets.
• Subjects – traditionally the dimension tables. Subjects are types of data that we might want to create a set of. For example Customers, Products, Suppliers, Contacts
• Lookups – tables that are used to lookup decodes, e.g. customer status, type, category.
• Converters – advanced metadata that tells Set Analysis how to convert a query from one subject to another. Converters are used for the most powerful types of sets.

Let’s look first at the database tables and database fields metadata, which is called Tables and Objects respectively.

Configuring Tables/Objects Metadata

We need to define the metadata for four tables in our example: CONTACTS, SECTORS, PRODUCTS and FACT.

CONTACT TABLE AND FIELDS

The first table we needed to set up is customer. The unique key for this table is Contact Id. Because the customer table is not a lookup table, we indicate that this is indeed a source table.

Most of this metadata can be imported automatically when you use the Load Tables and Load Fields button at the top of the interface.

Here is a screenshot of the completed definition. (Click on the image to enlarge)

I made only a few modifications to the default metadata that was imported using the import tables button. I added:

• to contact_id, the linked table DM Contacts
• to sector_id, the linked table DM Contact Sector
• a new field called Annual Income Band with the following formula (SQL Server Syntax):
  case when <a>.annual_income = 0 then ‘Unknown’ when <a>.annual_income < 30000 then ‘<30K’ when <a>.annual_income < 50000 then ‘30-50K’ when <a>.annual_income < 75000 then ‘50K-75K’ when <a>.annual_income < 125000 then ‘75K-125K’ else ‘125K+’ end
• a new field called Age Band with the following formula (SQL Server Syntax):
  case when <a>.age < 30 then ‘<30′ when <a>.age < 40 then ‘30-40′ when <a>.age < 50 then ‘40-50′ else ‘50+’ end

The new fields could have been defined within a database view (which is what I would recommend), but I wanted to demonstrate how they can also be added directly from within the metadata of Set Analysis. The syntax will be database specific. (There is much more information about this in the Set Architect Manual)

SECTOR TABLE AND FIELDS METADATA

The second table we needed to set up is sector. The unique key for this table is Sector Id. Because the sector table is a lookup table for customer, we indicate that this is indeed a Lookup Object and NOT a source table.

Here is a screenshot of the completed definition. (Click on the image to enlarge)

I made only one modifications to the default metadata that was imported using the import tables button. I added:

• to sector_id, the linked table DM Contact Sectors

As I’ve mentioned before you can define lookup tables from within Set Architect, but why? Why not create a new view and join the Contacts and Sectors tables into a common view on the sector_id? By pushing this to the database, you can simplify your configuration and you don’t have to worry about a lookup tables with lookups (e.g. state -> country -> continent).

PRODUCT METADATA

Next is the product table. The unique key for this table is Product Id. Because the product table is not a lookup table, we indicate that this is indeed a source table.

Here is a screenshot of the completed definition. (Click on the image to enlarge)

I made only one modifications to the default metadata that was imported using the import tables button. I added:

• to product_id, the linked table DM Products

FACT METADATA

The fourth and final table we needed to set up is fact table. There is no unique key for this fact table, but that doesn’t matter because we won’t be creating sets on top of orders. (If we needed to, then we would need an order_key which could be inserted into the set_ tables for linking). Because this fact table is not a lookup table, we indicate that this is a source table.

Here is a screenshot of the completed definition. (Click on the image to enlarge)

I made only a few modifications to the default metadata that was imported using the import tables button. I added:

• to contact_id, I added the linked table DM Contacts
• to product_id, I added the linked table DM Products
• a new field called Quarter with the formula (SQL Server Syntax):
  case when <a>.week between 1 and 13 then ‘Q1′ when <a>.week between 14 and 26 then ‘Q2′ when
  <a>.week between 27and 39 then ‘Q3′ else ‘Q4′ end

Keep in simple. In real life, if you wanted to add additional calculations like quarter or profit (sales_actual – cost), then do it in the database or through a database view. This will make your life easier when it comes to long term maintenance.

Once all the tables have been defined, we can move to Subjects.

DEFINING SUBJECTS

Subjects of a set are the central theme of the set. it will be associated with the table that most uniquely describes the subject for which sets are created. In our example, we are using contacts and products. We want to be able to create sets of both contacts and sets of products, so we will define two subjects.

Contact Subject

Here is how we will define the Contact Subject. (to enlarge click on picture)

Product Subject

Here is how we will define the Product Subject. (to enlarge click on picture)

Next, we need to check our lookups. This information should already be defined based on the changes we made when defining the sector table and defining it as a lookup for contacts.

DEFINING LOOKUPS

Here is how we will define the metadata for the sector lookup. (to enlarge click on picture)

DEFINING CONVERTERS

Converters are just plain cool. When I went through training on sets, I never really grasp what it is they do or their role in defining sets. Converters are used to allow you to convert from one subject to another. Clear as mud? Well, actually they allow you to define sets based on the “related” subject. Here are a couple of great examples:

• The set of customers that buy these products
• The set of products that have been purchased by these customers
• The set of other products sold with these products

So now you can track the set of Gold Customers who have purchased Premium Products during the period.

Converting Contacts to Products

The SQL we used for our converter was:
select distinct PRODUCT_ID as ID
from DW_CRM_FACT_ORDERS
where CONTACT_ID in ( <sub_query> )

Converting Products to Customers

The SQL we used for our converter was:
select distinct CONTACT_ID as ID
from DW_CRM_FACT_ORDERS
where PRODUCT_ID in ( <sub_query> )

The help text is completely optional and is not exposed to Set Analysis within the Performance Manager dashboard.

CONCLUSION

We’ve had an opportunity to walk through an example that covers the most common scenarios that I’ve seen in the field. If you have additional questions or comments, please list them below. I will use the database and metadata associated with this article to demonstrate the power of sets.

Read more…

So How Exactly Does Set Analysis Work?

Set Analysis works together with the tables that are already part of your Data Warehouse. For example, in your Data Warehouse today you have customers, products and orders, etc. Therefore all the tables that below to Set Analysis must reside in the same database as your datasource. (From this perspective, I like was SQL Server allows you to do. In SQL Server, as long as you specify the database.owner.table, you can have the database tables in physically different database instances. It works because they can share the same connection.)

Installing SetAnalyzer

The first thing you might have problems with is finding the Set Analyzer software. Although the engines for using Set Analyzer are built into the BusinessObjects Platform, you still need to install the SetArchitect legacy client-server software to do some minimal configuration.

Traditionally BusinessObjects provided SetArchitect as a seperate download; however starting in XI 3.0, it is located with your BusinessObjects XI 3.0 installation media under the Add-Ons directory. The actual file path is: D:\BOEXI30_Install_Files\win32_x86\Collaterals\Add-Ons\SetArchitect.

You may have trouble finding the documentation for the legacy Set Architect client application. The documentation will be installed when you install the Set Architect Add-on from the BusinessObjects XI 3.0 installation media. However if you want to see the documentation prior to the installation you can download it here:

• Set Architect Documentation
• Set Analyzer Documentation

NOTE: Traditionally Set Analyzer existed as a client-server application, but once it was integrated into the BusinessObjects Platform, it no longer became necessary to use Set Analyzer. Business Objects has moved all but some very basic configuration steps to the web-based Performance Manager interface.

Set Tables

The tables which belong to the Set Analysis application all belong with set_. There are just over 100 tables in all. These tables definitions get created one of two ways. The set_ tables are created when:

1. The Business Objects administrator creates the repository from the Dashboard and Analytics Setup Screen. From within System Setup\Repository, the administrator defines the repository connection and “Creates” the Performance Manager tables. When using this method, not only are the set tables created, but the tables for all the BusinessObjects Performance Manager Engines are created. (Standard Method)
   
2. The Set Analysis Administrator uses Set Analyzer Architect to “Setup” the repository. This method will only create the set tables for the Set Analyzer application. (Legacy Method – Not Recommended)

Once the Set Analysis tables have been created, you will be able to begin configuring your Set Analysis Data Sources. In a typical BusinessObjects Performance Manager Repository, this is what you would see:

Defining Your Data Sources

When setting up Set Analysis for the first time, there are only two tasks that need to be performed from within the legacy Set Architect application.

1. You will need to define a connection, so that Set Architect understands how to connect to the database which contains your data warehouse data and the set analyzer tables.
2. Next, you will need to set the metadata for your data warehouse tables so that set analyzer understands how to query the underlying database and create the sets.

Optimizing Your Data Sources and Set Analyzer

I highly recommend that if possible, you create database views on top of all the data sources that you plan to use for creating sets. For example if you have a customer table with 5 or 10 associated look-up tables for things such as geography, demographics, customer details, etc; then join these tables together and create a master view. Be aware that it is very important that your view NOT contain duplicate customer keys.

in Set Analyzer you need to define subjects. These are non-lookup tables associated with the fact table. In traditional data warehousing venacular they would be called dimension tables. These dimension tables need to be defined as Subjects in Set Architect.

If there is a many-to-many relationship between two tables, then I would need to create two subjects for each of the two tables. For example, if products have many suppliers and suppliers supply multiple products, (e.g. a many to many relationship exists), then I will need to define suppliers and products as Subjects. This will allow me to create complex sets such as:

• All suppliers who supply less than 10 products.
• All products which are supplied by suppliers in Georgia, who supply at least 5 products.

In my next post, I’ll provide screenshots of a sample metadata configuration as well as an excel spreadsheet for you to download and use yourself. All you will need to do is import the data into your favorite database.

Do you have an old leather jacket? You know the kind. The jacket that reminds you of the good old days. Each time to try it on, it fits so perfectly, you wonder why you don’t get it out more often. It’s tough, it’s cool, it’s unique, and everyone should have one. When I bring it out and show the kids – they love it.

Well, from a software perspective, my “old leather jacket” is Set Analysis. I recently had a situation where a customer needed the capabilities of Set Analysis in their organization to solve some specific reporting needs. I had to dust off the manuals and fire it back up again, but when I did, I wondered why we’d been apart so long.

Set Analysis is a product that was created by Next Action and was acquired by Business Objects way back in October of 1999. Business Objects then launched a company called Ithena, which created analytic application and use set analysis extensively. I provide this background to let you know that the Set Analysis product has been around for a long time and it still brings value today.

Set Analysis provides ability to categorize information. The fancy term for this is called segmentation and is typically market or customers segmentation (although it also can relate to products, employee, etc.) This capability is critical to help organizations understand how the customers and products are affecting their business. We use collections, or sets, to make lists of items to communicate and to manage.

In this Venn diagram you can see there are three sets:

• Last year’s gold customers
• Current silver customers
• Customers at risk

Segmentation products let Set Analysis make it easy to pick where you want to focus. For example the light green area represents the collection of: Last year’s gold customers who are at risk but are not currently silver.

Here is another example related to mortgage providers:

1. Customers who have a good credit rating
2. Customers who are paying an above average interest rate
3. Customers who have not refinanced in the last 24 months.
4. Customers who have never made a late payment.

Customers who exist in a set are called members.

Next, a mortgage company might take all four collections and market those customers who are members of all four sets with the goal of offering them a more competitive interest rate before they shop their mortgage with a competitor.

So what’s the big deal?

I hear what you are saying. What is the big deal? That’s just a normal query and you are right it is. What is facinating is the ability within a sets engine like Set Analysis to see the trends in these sets over time. Although it’s possible to get a snapshot of these collection members through a standard database query, what you don’t see is changes to the sets over time. For example you need to know:

1. How many Platinum Customers do I have? 1,000.
2. If I had 900 last month, did I have 100 join or did I have 200 new customers join and 100 leave?
3. Where did they go? Did they become Gold? Silver? Are they off the radar?

It’s fascinating to me to see how organizations are so focused on what they sell and yet there is so much value in looking at what you didn’t sell… and who you didn’t sell it to!

In a down economy like we see today, it seems even more important than ever before to understand what your customers are doing.

In the next few posts, I’ll take a little bit of a deeper dive into the product and share a few tricks and tips.