1月 222018
 

SAS/IML 14.3 (SAS 9.4M5) introduced a new syntax for creating lists and for assigning and extracting item in a list. Lists (introduced in SAS/IML 14.2) are data structures that are convenient for holding heterogeneous data. A single list can hold character matrices, numeric matrices, scalar values, and other lists, as discussed in a previous article about how to use lists in SAS/IML.

The list creation operator

You can use square brackets to create a list. The elements of the list are separated by commas. For example, the following syntax creates a list that contains three elements. The list represents a hypothetical patient in a cholesterol-lowering study.

proc iml;
/* 1. New list creation syntax (L = [val1, val2,...]) */
/*                 Cholesterol at
       Name  Age   0mo  3mo  6mo  12mo */
P1 = ["Bob",  36, {182, 170, 170, 162}]; /* Patient 1 */

The name of the list is P1. The first element of the list is a string, the second is a scalar number, and the third is a numeric vector. In this case, the elements are specified by using literal values, but you can also use previously defined variables. The following statement is equivalent but defines the list by using existing variables:

Name = "Bob"; Age = 36; Chol = {182, 170, 170, 162};
P1 = [Name, Age, Chol];   /* define list from copies of existing variables */

As mentioned earlier, lists can contain other lists. For example, if you have multiple patients in a study, you can create a list of each patient's data, then create a list that contains all the patients' data, as follows:

P2 = ["Fred", 52, {175, 165, 155}     ]; /* Patient 2 */
P3 = ["Tom",  45, {160, 145,   ., 139}]; /* Patient 3 */
Patients = [P1, P2, P3];                 /* a list of patients */

Assign and extract list items

You can use the list item operator ($) to specify an item in a list. For each patient, the age is stored as the second item in a list. If you want the age of Bob (the first patient), you can use either of the following statements:

/* 2. New list item syntax ($) */
BobAge = P1$2;         /* get 2nd item from P1 list */
BobAge = Patients$1$2; /* get 1st item of Patients, then 2nd item from that list */

The first statement is straightforward: P1$2 means "get the second item from the P1 list." The second statement is parsed left to right. The syntax Patient$1 means "get the first item from the Patients list," which is equivalent to P1. Thus Patient$1$2 gets Bob's age.

The preceding example uses literal values to specify the position of an item in a list, but you can also use a variable. This makes it possible to extract items in a loop. For example, the following statements loop over all patients, extract the name and cholesterol values of the patient, and compute each patient's average cholesterol value during the study:

N = ListLen(Patients);      /* number of patients */
Name = j(N,1,"     ");      /* allocate vector for names */
AvgChol = j(N,1,.);         /* allocate vector for results */
do i = 1 to N;
   Name[i] = Patients$i$1;  /* get name of i_th patient */
   Chol = Patients$i$3;     /* get sequence of cholesterol values */
   AvgChol[i] = mean(Chol); /* average value */
end;
print AvgChol[rowname=Name];

You can use the list item operator on the left side of an assignment operator to change an item in a list. For example, suppose you discover that Bob's age was typed wrong: Bob is really 63, not 36. You can update Bob's data as follows:

P1$2 = 63;         /* update 2nd item in P1 list */
Patients$1 = P1;   /* update entire patient list */

Alternatively, you could use the syntax Patients$1$2 = 63 to update the data in place.

Extract sublists

To subset a list, use square brackets ([]) and specify the indices of the items. For example, the following statement extracts the second and third patients into a new list:

/* 3. Extract sublist SL = L[ {i1, i2,...} ] */
SubList = Patients[ {2 3} ]; /* Fred and Tom */

The sublist operator ([]) ALWAYS returns a list, even if you specify only one item! Thus Patients[1] is a LIST that contains one item. If you want the item itself, use Patients$1.

Named lists

In the previous example, the items in the list P1 are a patient's name, age, and cholesterol readings. If you want to extract Bob's age, you can write P1$2, but someone unfamiliar with the order of the list items would have no idea what that item represents. Thus it is helpful to define a named list, which is also called an associative array. When you specify a named list, you specify the items by using name-value pairs, as follows:

P = [#"Name" = "Bob",                        /* P$1 or P$"Name" */
     #"Age"  = 63,                           /* P$2 or P$"Age" */
     #"Cholesterol" = {182, 170, 170, 162}]; /* P$3 or P$"Cholesterol" */

You can use the names to refer to the items in a list. For example, the following statements extract the patient's name and cholesterol readings by using the list item operator:

Name = P$"Name";
Chol = P$"Cholesterol";       /* get Bob's measurements */
print Chol[Label=Name];

You can also use names in the sublist operator to extract a sublist:

L = P[ {"Age" "Cholesterol"} ];  /* sublist that contains two items */

Summary

In summary, SAS/IML 14.3 contains new syntax for creating a list and for extracting items and sublists. This syntax makes it easier to use lists and to read and write SAS/IML programs that use lists.

The post Create lists by using a natural syntax in SAS/IML appeared first on The DO Loop.

1月 202018
 

SAS/GRAPH® Annotate FacilityThe

data myanno;                                                                                                                            
  length function color $8;                                                                                                                                                                                                                             
  function='move';                                                                                                                      
    x=0;  y=0;                                                                                                                          
    output;                                                                                                                             
  function='draw';                                                                                                                      
   x=100;  y=100;                                                                                                                       
   color='red';                                                                                                                         
   output;                                                                                                                              
run;                                                                                                                                    
 
proc gplot data=sashelp.cars;                                                                                                           
  plot mpg_highway*cylinders / vaxis=axis1 haxis=axis2 annotate=myanno;                                                                                         
  symbol1 interpol=none value=dot color=blue; 
  axis1 label=(angle=90);                                                                                                               
  axis2 offset=(2,2)pct;                                                                                                                                                                                                          
run;                                                                                                                                    
quit;

 

The following annotate errors are written to the SAS log when I run this code:

NOTE: ERROR DETECTED IN ANNOTATE= DATASET WORK.MYANNO.
NOTE: PROBLEM IN OBSERVATION     2 -
      A CALCULATED COORDINATE LIES OUTSIDE THE VISIBLE AREA           X
      A CALCULATED COORDINATE LIES OUTSIDE THE VISIBLE AREA           Y

 

Here is the resulting graph:

The annotated line is drawn outside the axis area. But why? I defined my X and Y coordinates for the MOVE and DRAW functions correctly, did I not?

The coordinates are defined correctly, but what I did not define is the coordinate system for the annotation. The XSYS and

data myanno;                                                                                                                            
  length function color $8;                                                                                                             
  retain xsys ysys '1';                                                                                                                 
  function='move';                                                                                                                      
    x=0;  y=0;                                                                                                                          
    output;                                                                                                                             
  function='draw';                                                                                                                      
   x=100;  y=100;                                                                                                                       
   color='red';                                                                                                                         
   output;                                                                                                                              
run;                                                                                                                                    
 
proc gplot data=sashelp.cars;                                                                                                           
  plot mpg_highway*cylinders / vaxis=axis1 haxis=axis2 annotate=myanno;                                                                                         
  symbol1 interpol=none value=dot color=blue;
  axis1 label=(angle=90);                                                                                                               
  axis2 offset=(2,2)pct;                                                                                                                                                                                                           
run;                                                                                                                                    
quit;

 

Here is the graph containing the correct line:

Creating Multiple Graphs with an Annotate Data Set, BY-and-BY

The need to generate multiple graphs from one procedure using a BY statement is very common. However, using an Annotate data set with a BY statement can be a little tricky. Here are the general rules for using an Annotate data set with a SAS/GRAPH procedure that creates multiple graphs with a BY statement:

  1. Make sure that the Annotate data set and the input data set for the procedure include the same BY variables. The BY variables must also be the same data type in both data sets.
  2. Both the Annotate data set and the input data set must be sorted by the BY variables.
  3. Include the ANNOTATE= (or ANNO=) option in the action statement of the SAS/GRAPH procedure.

The goal of the following program is to create two graphs using a BY statement in which the annotation is specific to each graph. The Annotate data set draws the maximum MPG_Highway value at the maximum point for each X value.

/* Compute the maximum MPG_Highway values */                                                                                            
proc sort data=sashelp.cars(where=(origin in('USA' 'Europe'))) out=cars;                                                                
  by origin cylinders;                                                                                                                  
run;                                                                                                                                    
 
proc means data=cars noprint;                                                                                                           
  by origin cylinders;                                                                                                                  
  var mpg_highway;                                                                                                                      
  output out=meansout max=max;                                                                                                          
run;                                                                                                                                    
 
data myanno;                                                                                                                            
  length function color text $8;                                                                                                        
  retain xsys ysys '2' color 'black' position '2' size 1.5;                                                                             
  set meansout;                                                                                                                         
 
  function='label';                                                                                                                     
    x=cylinders;  y=max;                                                                                                                
    text=strip(max);                                                                                                                    
    output;                                                                                                                             
run;                                                                                                                                    
 
proc gplot data=cars annotate=myanno;                                                                                                   
  by origin;                                                                                                                            
  plot mpg_highway*cylinders / vaxis=axis1 haxis=axis2;                                                                                 
  symbol1 interpol=none value=dot color=blue;                                                                                           
  axis1 label=(angle=90);                                                                                                               
  axis2 offset=(2,2)pct;                                                                                                                
run;                                                                                                                                    
quit;

 

Here are the resulting graphs:

There are two issues here. First, there should be only one maximum value displayed for each X value. There are duplicate values of the annotated text on each graph. Second, the following messages are written to the SAS log:

NOTE: ERROR DETECTED IN ANNOTATE= DATASET WORK.MYANNO.
NOTE: PROBLEM IN OBSERVATION     1 -
      DATA SYSTEM REQUESTED, BUT VALUE IS NOT ON GRAPH    'Y'
NOTE: PROBLEM IN OBSERVATION     5 -
      DATA SYSTEM REQUESTED, BUT VALUE IS NOT ON GRAPH    'X'
NOTE: The above message was for the following BY group:
      Origin=USA

 

These notes tell me that either the X or the Y coordinate in two of the observations in the Annotate data set do not exist on one of the graphs. This issue occurs because the Annotate coordinates for each of the BY values are different for each graph. The axis ranges are different on the two graphs. So, when all of the annotation, instead of the annotation for only each BY value, is drawn on each graph, some of the Annotate coordinates cannot be found on the graph.

Both of these issues occur because the ANNOTATE=Myanno option is in the PROC GPLOT statement instead of in the action (PLOT) statement. Moving the ANNOTATE=Myanno option to the PLOT statement generates the expected output:

proc gplot data=cars;                                                                                                                   
  by origin;                                                                                                                            
  plot mpg_highway*cylinders / vaxis=axis1 haxis=axis2 annotate=myanno;                                                                 
  symbol1 interpol=none value=dot color=blue;                                                                                           
  axis1 label=(angle=90);                                                                                                               
  axis2 offset=(2,2)pct;                                                                                                                
run;                                                                                                                                    
quit;

 

Off the Grid

Another common issue with using an Annotate data set is when a coordinate in the Annotate data set lies outside the range of an axis on the graph. For example, I will chart the mean MPG_Highway values with the GCHART procedure and draw a symbol at the maximum value for each country of origin using an Annotate data set:

proc sort data=sashelp.cars out=cars;                                                                                                   
  by origin;                                                                                                                            
run;                                                                                                                                    
 
/* Compute the mean and the max */                                                                                                      
proc means data=cars noprint;                                                                                                           
  by origin;                                                                                                                            
  var mpg_highway;                                                                                                                      
  output out=meansout mean=mean max=max;                                                                                                
run;                                                                                                                                    
 
data myanno;                                                                                                                            
  length function color $8 text $14;                                                                                                    
  retain xsys ysys '2' color 'red' position '2' size 2;                                                                                 
  set meansout;                                                                                                                         
 
  function='symbol';                                                                                                                    
    midpoint=origin;  y=max;                                                                                                            
    text='diamondfilled';                                                                                                               
    output;                                                                                                                             
run;                                                                                                                                    
 
proc gchart data=meansout;                                                                                                              
  vbar origin / sumvar=mean annotate=myanno raxis=axis1;                                                                                
  axis1 label=(angle=90);                                                                                                               
run;                                                                                                                                    
quit;

 

When I run this program, the following graph is produced, excluding the annotated symbols:

The following annotate error messages are written to the SAS log:

NOTE: ERROR DETECTED IN ANNOTATE= DATASET WORK.MYANNO.
NOTE: PROBLEM IN OBSERVATION     1 -
      DATA SYSTEM REQUESTED, BUT VALUE IS NOT ON GRAPH    'RESPONSE'
NOTE: PROBLEM IN OBSERVATION     2 -
      DATA SYSTEM REQUESTED, BUT VALUE IS NOT ON GRAPH    'RESPONSE'
NOTE: PROBLEM IN OBSERVATION     3 -
      DATA SYSTEM REQUESTED, BUT VALUE IS NOT ON GRAPH    'RESPONSE'

 

These messages tell me that multiple response values (Y coordinates) in the Annotate data set lie outside the range of the Y axis. The procedure does not automatically extend the Y-axis range to accommodate the annotation, so I need to do this by including the ORDER= option in the AXIS1 statement:

proc gchart data=meansout;                                                                                                              
  vbar origin / sumvar=mean annotate=myanno raxis=axis1;                                                                                
  axis1 label=(angle=90) order=(0 to 70 by 10);                                                                                         
run;                                                                                                                                    
quit;

 

The correct graph is now generated:

Annotation is a useful tool that enables you to draw features on a graph that the graphics procedure might not have the capability to draw. Using an Annotate data set is easier once you understand what the SAS log messages are telling you and can take steps to avoid common issues. Don’t be afraid to dive in!

Happy drawing!

Common annotate pitfalls and how to avoid them was published on SAS Users.

1月 192018
 

Technology is changing rapidly: autonomous vehicles, connected devices, digital transformation, the Internet of Things (IoT), machine learning, artificial intelligence (AI), automation. The list goes on. And it has only begun. I do not try to predict the future. Instead, I examine the trends in technology and look for disruptive forces [...]

Two tech trends shaping 2018 and beyond was published on SAS Voices by Oliver Schabenberger

1月 182018
 

One of the most exciting features from the newest release of Visual Data Mining and Machine Learning on SAS Viya is the ability to perform Market Basket Analysis on large amounts of transactional data. Market Basket Analysis allows companies to analyze large transactional files to identify significant relationships between items. While most commonly used by retailers, this technique can be used by any company that has transactional data.

For this example, we will be looking at customer supermarket purchases over the past month. Customer is the Transaction ID; Time is the time of purchase; and Product is the item purchased. The data must be transactional in nature and not aggregated, with one row for each product purchased by each customer.

Market Basket Analysis in SAS Viya

With our data ready, we can now perform the analysis using the MBANALYSIS Procedure. As illustrated below in SAS Studio, by specifying pctsupport=1, we will only look at items, or groups of items, that appear in at least 1% of the transactions. For very large datasets this saves time by only looking at combinations of items that appear frequently. This allows extraction of the most common and most useful relationships.

The MBANALYSIS procedure outputs a list of significant relationships, called Association Rules, by calculating the LIFT metric. A lift greater than one generally indicates that a Rule is significant. By default, each relationship has two items, although this can be changed to include multiple items.

Below is a screenshot of the ten most important rules. The first item in the rule is the “Left Hand Side” and the second item after the arrow is the “Right Hand Side.” For the first rule, we can see that coke and ice cream appear together in 220 transactions and have a lift of 2.37, meaning purchasing Coke makes the purchase of ice cream about twice as likely.

Top 10 Association Rules

While Association Rules above give powerful insights into large transactional datasets, the challenge is exploring these rules visually. One way to do this is by linking the rules together via a Network Diagram. This allows users to see the relationships between multiple rules, and identify the most important items in the network. The following SQL code prepares the data for the Network Diagram.

Network Diagrams plot a set of “Source” values (T1_ITEM), and connects them to a “Target” value (ITEM2). If the source value represents the left hand side of the rule, the corresponding right hand side of the rule is listed as the Target variable. We will use the “Lift” value to link these source and target variables. If the target value is the right hand side of the rule, the target and the lift are missing. This allows us to plot the product, but no linkage will be made.

Now, my data is ready to be visualized as a Network Diagram. Using the following code, I am able to promote my Association Rules, making this dataset available via SAS® Visual Analytics.

Now, I am able to quickly and easily generate my Network Diagram without having to create any code.

Hovering over a node allows me to see specific information about that particular item. Here, we can see that Heineken was purchased in 59.9% of all transactions, which is 600 transactions.

Hovering over the linkage, we can see specific information about the rule. Below, we can see that purchasing artichoke (artichoke) makes the purchase of Heineken about 38% more likely. Many other rules link to Heineken, showing its importance in the network. Business Unit Experts can use this diagram as a starting point to analyze selling strategies to make proper adjustments for the business.

Conclusion

The Market Basket Analysis procedure in Visual Data Mining and Machine Learning on SAS Viya can help retailers quickly scan large transactional files and identify key relationships. These relationships can then be visualized in a Network Diagram to quickly and easily find important relationships in the network, not just a set of rules. As transactional data, whether in-store, online, or in any other form gets bigger, this Market Basket functionality is a must have weapon in the analytical toolkit of any business.

Visualizing the results of a Market Basket Analysis in SAS Viya was published on SAS Users.

1月 172018
 

Wherever there is uncertainty there has got to be judgment, and wherever there is judgment there is an opportunity for human fallibility. Donald Redelmeirer, physician-researcher Over the holidays, I read a fascinating book titled The Undoing Project: A Friendship That Changed Our Mind by Michael Lewis (W.W. Norton & Company, [...]

Why do we rely on judgment when analytics outperforms it? was published on SAS Voices by Charlie Chase

1月 172018
 

Wherever there is uncertainty there has got to be judgment, and wherever there is judgment there is an opportunity for human fallibility. Donald Redelmeirer, physician-researcher Over the holidays, I read a fascinating book titled The Undoing Project: A Friendship That Changed Our Mind by Michael Lewis (W.W. Norton & Company, [...]

Why do we rely on judgment when analytics outperforms it? was published on SAS Voices by Charlie Chase

1月 172018
 

Money magazine (Jan/Feb 2018) contains an article about how much it costs to give birth in the US. The costs, which are based on insurance data, include prenatal care and hospital delivery but exclude infant care. The data are compiled for each state (including Washington, DC) and by type of delivery (vaginal versus cesarean section). The data includes the average and median costs for each state.

The online version of the article contains a map and a table of average costs, colored by the quintiles of the costs. Because I think that median costs are more relevant, I decided to create a visualization of the distribution of the median costs. Additionally, I want to visualize the incremental cost of a C-section over a vaginal delivery. According to the CDC, about 32% of deliveries are C-sections in the US. Cesarean delivery is major surgery and often requires an additional two days of hospital recovery in addition to operating-room charges.

With a little sleuthing, I was able to locate the data and download it into a SAS data set. You can download the data and SAS program that creates the graphs in this article.

Cost Distribution and incremental cost of a cesarean delivery

Median cost of vaginal and cesarean delivery by US state (2016-2017)

The adjacent bar chart (click to enlarge) shows the distribution of the median costs of childbirth in the US. Since the median cost of a cesarian delivery is always more than the median cost of a vaginal delivery, I overlaid the two graphs. The states are ordered by the median cost of a vaginal delivery. The data shows that the states of Alabama, Rhode Island, Nebraska, Louisiana, and Utah are the least expensive states for vaginal delivery. The median cost is about $5000 in those states. The most expensive states include Alaska, New Jersey, New York, Wisconsin, and Massachusetts. The median cost is more than $8000 for those states, with Alaska topping out at $14,500.

If you are more interested in the cost of a cesarean delivery, I created a similar graph sorted by the cost of a C-section. No matter how you sort it, the graph indicates that a C-section costs about $2500 to $3500 more than a vaginal delivery. In Washington, DC, the incremental cost is about $1100, which is relatively low. In Vermont and Alaska, the incremental cost is more than $4000, which is relatively high.

The Money magazine map of the data does not reveal any unexpected regional trends. Costs are high in Alaska and New England. Costs are low in some southern states.

Comparing delivery types by using a scatter plot

For a more sophisticated audience, you can use a scatter plot to plot the costs for vaginal and cesarean delivery in each state. A plot of the median costs is shown below. A regression line to these data has a slope of 1.2, which indicates that, on average, the median cost of a C-section is about 20% more than for a vaginal delivery. This visualization also enables you to see that Alaska is an extreme outlier for both types of delivery.

Median cost of vaginal and cesarean delivery by US state (2016-2017)

The Money article about these data points out two facts that cannot be seen in the data. First, it says that women "who have no insurance... are usually charged a higher amount than the negotiated rate." Second, US women "pay more to have a baby than residents of any other country. The highest prices in the U.S. were more than double those of the second-most expensive country, Switzerland" (emphasis added)." For a comparison of different countries, see Parents magazine (Jan 2017).

What interesting facts do you notice about these data? Leave a comment.

The post How much does it cost to give birth in the US? appeared first on The DO Loop.