5月 292018
 

The SAS language provides syntax that enables you to quickly specify a list of variables. SAS statements that accept variable lists include the KEEP and DROP statements, the ARRAY statement, and the OF operator for comma-separated arguments to some functions. You can also use variable lists on the VAR statements and MODEL statements of analytic procedures.

This article describes six ways to specify a list of variables in SAS. There is a section in the SAS documentation that describes how to construct lists, but this blog post provides more context and a cut-and-paste example for every syntax. This article demonstrates the following:

  • Use the _NUMERIC_, _CHARACTER_, and _ALL_ keywords to specify variables of a certain type (numeric or character) or all types.
  • Use a single hyphen (-) to specify a range of variables that have a common prefix and a sequential set of numerical suffixes.
  • Use the colon operator (:) to specify a list of variables that begin with a common prefix.
  • Use a double-hyphen (--) to specify a consecutive set of variables, regardless of type. You can also use a variation of this syntax to specify a consecutive set of variables of a certain type (numeric or character).
  • Use the OF operator to specify variables in an array or in a function call.
  • Use macro variables to specify variables that satisfy certain characteristics.

Some companies might discourage the use of variable lists in production code because automated lists can be volatile. If the number and names of variables in your data sets occasionally change, it is safer to manually list the variables that you are analyzing. However, for developing code and constructing examples, lists can be a huge time saver.

Use the _NUMERIC_, _CHARACTER_, and _ALL_ keywords

You can specify all numeric variables in a data set by using the _NUMERIC_ keyword. You can specify all character variables by using the _CHARACTER_ keyword. Many SAS procedures use a VAR statement to specify the variables to be analyzed. When you want to analyze all variables of a certain type, you can use these keywords, as follows:

/* compute descriptive statistics of allnumeric variables */
proc means data=Sashelp.Heart nolabels; 
   var _NUMERIC_;          /* _NUMERIC_ is the default */
run;
 
/* display the frequencies of all levels for all character variables */
proc freq data=Sashelp.Heart; 
   tables _CHARACTER_;    /* _ALL_ is the defaul */
run;
Use a keyword to specify a list of variables in SAS

One of my favorite SAS programming tricks is to use these keywords in a KEEP or DROP statement (or data set option). For example, the following statements create a new data set that contains all numeric variables and two character variables from the Sashelp.Heart data:

data HeartNumeric;
set Sashelp.Heart(keep=_NUMERIC_            /* all numeric variables */
                       Sex Smoking_Status); /* two character variables */
run;

An example of using the _ALL_ keyword is shown in the section that discusses the OF operator.

Use a hyphen to specify numerical suffixes

In many situations, variables are named with a common prefix and numerical suffix. For example, financial data might have variables that are named Sales2008, Sales2009, ..., Sales2017. In simulation studies, variables often have names such as X1, X2, ..., X50. The hyphen enables you to specify the first and last variable in a list. The first example can be specified as Sales2008-Sales2017. The second example is X1-X50.

The following DATA step creates 10 variables, including the variables x1-x6. Notice that the data set variables are not in alphanumeric order. That is okay. The syntax x1-x6 will select the six variables x1, x2, x3, x4, x5, and x6 regardless of their physical order in the data. The call to PROC REG uses the six variables in a linear regression:

data A;
   retain Y x1 x3 Z x6 x5 x2 W x4 R;  /* create 10 variables and one observation. Initialize to 0 */
run;
proc reg data=A plots=none;
   model Y = x1-x6;
run;

The parameter estimates from PROC REG are displayed in the order that you specify in the MODEL statement. However, if you use the SET statement in a DATA step, the variables appear in the original order unless you intentionally reorder the variables:

data B;
   set A(keep=x1-x6);
run;

Use the colon operator to specify a prefix

If you want to use variables that have a common prefix but have a variety of suffixes, you can use the colon operator (:), which is a wildcard character that matches any name that begins with a specified prefix. For example, the following DATA step creates a data set that contains 10 variables, including five variables that begin with the prefix 'Sales'. The subsequent DATA step drops the variables that begin with the prefix 'Sales':

data A;
retain Sales17 Y Sales16 Z SalesRegion Sales_new Sales1 R; /* 1 obs. Initialize to 0 */
run;
 
data B;
   set A(drop= Sales: ); /* drop all variables that begin with 'Sales' */
run;

Use a double-hyphen to specify consecutive variables

The previous sections used wildcard characters to match variables that had a specified type or prefix. In the previous sections, you will get the same set of variables regardless of how they might be ordered in the data set. You can use a double-hyphen (--) to specify a consecutive set of variables. The variables you get depend on the order of the variables in the data set.

data A;
   retain Y 0   x3 2   C1 'A'   C2 'BC'
          Z 3   W  4   C4 'D'   C5 'EF'; /* Initialize eight variables */
run;
data B;
   set A(keep=x3--C4);
run;

In this example, the data set B contains the variables x3, C1, C2, Z, W, and C4. If you use the double-hyphen to specify a list, be sure that you know the order of the variables and that this order is never going to change. If the order of the variables changes, your program will behave differently.

You can also specify all variables of a certain type within a range of variables. The syntax Y-numeric-Z specifies all numeric variables between Y and Z in the data set. The syntax Y-character-Z specifies all character variables between Y and Z. For example, the following call to PROC CONTENTS displays the variables (in order) in the Sashelp.Heart data. The call to PROC LOGISTIC specifies all the numeric variables between (and including) the AgeCHDiag variable and the Smoking variable:

proc contents data=Sashelp.Heart order=varnum ;
run;
 
proc logistic data=Sashelp.Heart;
   model status = AgeCHDdiag-numeric-Smoking;
   ods select ParameterEstimates;
run;
Use a double-hyphen to specify a contiguous list of variables in SAS

Arrays and the OF operator

You can use variable lists to assign an array in a SAS DATA step. For example, the following program creates a numerical array named X and a character array named C. The program finds the maximum value in each row and puts that value into the variable named rowMaxNUm. The program also creates a variable named Str that contains the concatenation of the character values for each row:

data Arrays;
   set sashelp.Class;
   array X {*} _NUMERIC_;        /* X[1] is 1st var, X[2] is 2nd var, etc */
   array C {*} _CHARACTER_;      /* C[1] is 1st var, C[2] is 2nd var, etc */
   /* use the OF operator to pass values in array to functions */
   rowMaxNum = max(of x[*]);     /* find the max value in this array (row) */
   length Str $30;
   call catx(' ', Str, of C[*]); /* concatenate the strings in this array (row) */
   keep rowMaxNum Str;
run;
 
proc print data=Arrays(obs=4);
run;
Use a keyword to specify a list of variables to certain SAS functions

You can use the OF operator directly in functions without creating an array. For example, the following program uses the _ALL_ keyword to output the "complete cases" for the Sashelp.Heart data. The program drops any observation that has a missing value for any variable:

data CompleteCases;
  set Sashelp.Heart;
  if cmiss(of _ALL_)=0;  /* output only complete cases for all vars */
run;

Use macro variables to specify a list

The previous sections demonstrate how you can use syntax to specify a list of variables to SAS statements. In contrast, this section describes a technique rather than syntax. It is sometimes the case that the names of variables are in a column in a data set. There might be other columns in the data set that contain characteristics or statistics for the variables. For example, the following call to PROC MEANS creates an output data set (called MissingValues) that contains columns named Variable and NMiss.

proc means data=Sashelp.Heart nolabels NMISS stackodsoutput;
   var _NUMERIC_;
   ods output Summary = MissingValues;
run;
proc print; run;
Use a macro variable to specify a list of variables in SAS

Suppose you want to keep or drop those variables that have one or more missing values. The following PROC SQL call creates a macro variable (called MissingVarList) that contains a space-separated list of all variables that have at least one missing value. This technique has many applications and is very powerful.

/* Use PROC SQL to create a macro variable (MissingVarList) that contains
   the list of variables that have a property such as missing values */
proc sql noprint;                              
 select Variable into :MissingVarList separated by ' '
 from MissingValues
 where NMiss > 0;
quit;
%put &=MissingVarList;
MISSINGVARLIST=AgeCHDdiag Height Weight MRW Smoking AgeAtDeath Cholesterol

You can now use the macro variable in a KEEP, DROP, VAR, or MODEL statement, such as KEEP=&MissingVarList;

Summary

This article shows six ways to specify a list of variables to SAS statements and functions. The SAS syntax provides keywords (_NUMERIC_, _CHARACTER_, and _ALL_) and operators (hyphen, colon, and double-hyphen) to make it easy to specify a list of variables. You can use the syntax in conjunction with the OF operator to pass a variable list to some SAS functions. Lastly, if the names of variables are stored in a column in a data set, you can use the full power of PROC SQL to create a macro variable that contains variables that satisfy certain criteria.

Do you use shorthand syntax to specify lists of variables? Why or why not? Leave a comment.

The post 6 easy ways to specify a list of variables in SAS appeared first on The DO Loop.

5月 252018
 

Have you ever tried to plot data on a map of Antarctica ... and been thoroughly frustrated or confused?!? If you're that person, or even a seasoned map maker wanting to hone your skills, then this blog post is for you! But first, here is a picture to get you [...]

The post Plotting data on Antarctica - a mapping challenge! appeared first on SAS Learning Post.

5月 242018
 

Poverty. It's a bit difficult to define who lives in poverty - I guess it's a relative thing, and depends on the standard of living of the people around you. Today we're going to take a look at the child poverty rates in several 'rich' countries (such as the United [...]

The post Comparing child poverty in 26 rich countries appeared first on SAS Learning Post.

5月 232018
 

In August 2017, Britta Gross spoke about General Motors’ perspective on bringing electric vehicles (and their derivatives) to market. Her point of view reaffirmed GM's research on consumer awareness of electric vehicles (only 60 percent) and consumer adoption concerns with this emerging technology. She also revealed the portfolio of cars [...]

Could analytics improve electric vehicle adoption? was published on SAS Voices by Lonnie Miller

5月 232018
 

Path analysis is an exploration of a chain of consecutive events that a given user or cohort performs during a set period while using a website, online game or mobile app (although other use cases can apply outside of digital analytics). As a subset of behavioral analytics, path analysis is [...]

SAS Customer Intelligence 360: Path analysis for re-engagement was published on Customer Intelligence Blog.

5月 232018
 
Butterfly plot of cholesterol by gender in SAS

This article shows how to construct a butterfly plot in SAS. A butterfly plot (also called a butterfly chart) is a comparative bar chart or histogram that displays the distribution of a variable for two subpopulations. A butterfly plot for the cholesterol readings of 5,057 patients in a medical study is shown to the right, where the distribution for the males is shown on the left side of the plot and the distribution for females is displayed on the right. (Click to enlarge.) The main contribution of this article is showing how to bin a continuous variable in SAS to form a butterfly chart.

The butterfly plot is similar to a comparative histogram because both enable you to compare the distribution of a continuous variable for subpopulations. The comparative histogram uses a panel to visualize several subpopulations in a panel, where each row represents a level of a classification variable. In contrast, the butterfly plot is limited to two levels and displays the distributions back-to-back.

Bin a continuous variable for each classification level

In a previous blog post, I constructed a butterfly chart that compares voice versus text usage by decade of age for cell phones. Similarly, there is a SAS Sample that shows how to create a butterfly chart for types of cancers by gender. In both of these examples, the butterfly chart is a comparative bar chart because the distribution shown is for a discrete variable ("decade of age" or "type of cancer"). This section shows how to start with a continuous variable (cholesterol) and bin it into intervals. You can then use the previous techniques to visualize the counts in each interval for each gender.

You can bin a continuous variable by using the BIN and TABULATE functions in SAS/IML or by using the OUTHIST= option on the HISTOGRAM statement in PROC UNIVARIATE. The following statements create an output data set (OutHist) that contains the counts of males and females that have cholesterol reading within bins of width 20. The bin width and the centers of the intervals are chosen automatically by PROC UNIVARIATE, but you can use the MIDPOINTS= option (shown in the comments) to control the placement of the intervals.

proc univariate data=Sashelp.Heart;
   class Sex;
   var Cholesterol;
   histogram cholesterol / nrows=2 outhist=OutHist 
                          /* midpoints=(80 to 560 by 40) */ /* control bin widths and locations */
                           odstitle="Cholesterol by Gender";
   ods select histogram;
run;
Comparative histogram in SAS: Cholesterol by Gender

Create a butterfly plot in SAS

The OutHist data set is in "long form." You need to convert it to "wide form" in order to construct a butterfly plot. The SAS code performs the following tasks:

  • Use a DATA step and WHERE clauses to convert the data from long to wide format.
  • Multiply the counts for the males by -1. The negative counts will be plotted on the left side of the butterfly plot.
  • Define a format that will display the absolute values of the counts. The axis for the formatted variables contains zero in the middle and increases in both directions.
  • Use the HBAR statements to plot the back-to-back bar charts for males and females.
/* convert data from long format to wide format */
data Butterfly;
   keep Cholesterol Males Females;
   label Males= Females= Cholesterol=; /* remove labels */
   merge OutHist(where=(sex="Female") rename=(_COUNT_=Females _MIDPT_=Cholesterol))
         OutHist(where=(sex="Male")   rename=(_COUNT_=Males   _MIDPT_=Cholesterol));
   by Cholesterol;
   Males = -Males;                     /* trick: reverse the direction of male counts */
run;
 
/* define format that displays the absolute value of a number */
proc format;
   picture positive low-<0="000,000"
   0<-high="000,000";
run;
 
ods graphics / reset;
title "Butterfly Plot of Cholesterol Counts By Gender";
proc sgplot data=Butterfly;
   format Males Females positive.;
   hbar Cholesterol / response=Males   legendlabel="Males";
   hbar Cholesterol / response=Females legendlabel="Females";
   xaxis label="Count" grid 
         min=-520 max=520 values=(-500 to 500 by 100) valueshint;
   yaxis label="Cholesterol" discreteorder=data;
run;

The graph is shown at the top of this article. You can see that the mode of the distribution is higher for males, and the distribution for males also has a longer tail.

A butterfly fringe plot

The butterfly plot is usually displayed with horizontal bars, as shown. However, you could use the VBAR statement to get a rotated version of the butterfly plot. As mentioned earlier, you can also use the MIDPOINTS= option on the HISTOGRAM statement to change the width of the histogram bins.

One useful variation in the butterfly plot is to use a very small bin width and replace the bar chart with a high-low plot. This creates a graph that I call a butterfly fringe plot. Recall that the usual fringe plot (also called a "rug plot") places a tick mark on an axis to show the distribution of data values. A fringe plot can suffer from overplotting when more than one observation has the same value. With a butterfly fringe plot, some ticks are higher than others (to represent repeated values) and bars that point up represent one binary value whereas bars that point down represent the other. The butterfly fringe plot provides a more complete visualization of the distribution of data for two levels of a response or classification variable.

ods graphics / width=640px height=180;
title "Butterfly Fringe Plot: Cholesterol Counts By Gender";
proc sgplot data=Butterfly;
   format Males Females positive.;
   highlow x=Cholesterol low=Males high=Females;
   refline 0 / axis=y;
   inset "Males"   / position=BottomLeft;
   inset "Females" / position=TopLeft;
   yaxis label="Count" grid;
   xaxis label="Cholesterol" discreteorder=data;
run;
Butterfly fringe plot of cholesterol by gender in SAS

The butterfly fringe plot was created by using a bin width of 5 for the cholesterol variable. That is, midpoints=(50 to 560 by 5).

In summary, this article shows how to create a butterfly plot for a continuous variable and a binary classification variable. When you bin the continuous variable, you obtain counts for each interval. You can then graph the counts back-to-back to form the butterfly chart. An interesting variation is the butterfly fringe plot, which combines a butterfly chart and a fringe plot.

The post A butterfly plot for comparing distributions appeared first on The DO Loop.

5月 222018
 

SAS ViyaSAS Viya Presentations is our latest extension of the SAS Platform and interoperable with SAS® 9.4. Designed to enable analytics to the enterprise, it seamlessly scales for data of any size, type, speed and complexity. It was also a star at this year’s SAS Global Forum 2018. In this series of articles, we will review several of the most interesting SAS Viya talks from the event. Our first installment reviews Hadley Christoffels’ talk, A Need For Speed: Loading Data via the Cloud.

You can read all the articles in this series or check out the individual interviews by clicking on the titles below:
Part 1: Technology that gets the most from the Cloud.


Technology that gets the most from the Cloud

Few would argue about the value the effective use of data can bring an organization. Advancements in analytics, particularly in areas like artificial intelligence and machine learning, allow organizations to analyze more complex data and deliver faster, more accurate results.

However, in his SAS Global Forum 2018 paper, A Need For Speed: Loading Data via the Cloud, Hadley Christoffels, CEO of Boemska, reminded the audience that 80% of an analyst’s time is still spent on the data. Getting insight from your data is where the magic happens, but the real value of powerful analytical methods like artificial intelligence and machine learning can only be realized when “you shorten the load cycle the quicker you get to value.”

Data Management is critical and still the most common area of investment in analytical software, making data management a primary responsibility of today’s data scientist. “Before you can get to any value the data has to be collected, has to be transformed, has to be enriched, has to be cleansed and has to be loaded before it can be consumed.”

Benefits of cloud adoption

The cloud can help, to a degree. According to Christoffels, “cloud adoption has become a strategic imperative for enterprises.” The advantages of moving to a cloud architecture are many, but the two greatest are elasticity and scalability.

Elasticity, defined by Christoffels, allows you to dynamically provision or remove virtual machines (VM), while scalability refers to increasing or decreasing capacity within existing infrastructure by scaling vertically, moving the workload to a bigger or smaller VM, or horizontally, by provisioning additional VM’s and distributing the application load between them.

“I can stand up VMs in a matter of seconds, I can add more servers when I need it, I can get a bigger one when I need it and a smaller one when I don’t, but, especially when it comes to horizontal scaling, you need technology that can make the most of it.” Cloud-readiness and multi-threaded processing make SAS® Viya® the perfect tool to take advantage of the benefits of “clouding up.”

SAS® Viya® can addresses complex analytical challenges and speed up data management processes. “If you have software that can only run on a single instance, then scaling horizontally means nothing to you because you can’t make use of that multi-threaded, parallel environment. SAS Viya is one of those technologies,” Christoffels said.

Challenges you need to consider

According to Christoffels, it’s important, when moving your processing to the cloud, that you understand and address existing performance challenges and whether it will meet your business needs in an agile manner. Inefficiencies on-premise are annoying; inefficiencies in the cloud are annoying and costly, since you pay for that resource.

It’s not the best use of the architecture to take what you have on premise and just shift it. “Finding and improving and eliminating inefficiencies is a massive part in cutting down the time data takes to load.”

Boemska, Christoffels’ company, has tools to help businesses find inefficiencies and understand the impact users have on the environment, including:

  1. Real-time diagnostics looking at CPU Usage, Memory Usage, SAS Workload, etc.
  2. Insight and comparison provides a historic view in a certain timeframe, essential when trying to optimize and shave off costly time when working in cloud.
  3. Utilization reports to better understand how the platform is used.

Optimizing inefficiencies with SAS Viya

But scaling vertically and horizontally from cloud-based infrastructure to speed the loading and data management process solves only part of the problem. Christoffels said SAS Viya capabilities completes the picture. SAS Viya offers a number of benefits in a Cloud infrastructure, Christoffels said. Code amendments that make use of the new techniques and benefits now available in SAS Viya, such as the multi-threaded DATA step or CAS Action Sets, can be extremely powerful.

One simple example of the benefits of SAS Viya, Christoffels said, is that with in-memory processing, PROC SORT is a procedure that’s no longer needed; SAS Viya does “grouping on the fly,” meaning you can remove sort routines from existing programs, which of itself, can cut down processing time significantly.

As a SAS Programmer, just the fact that SAS Viya can run multithreaded, the fact that you don’t have to do these sorts, the way it handles grouping on the fly, the fact that multithreaded nature and capability is built into how you deal with tables are all “significant,” according to Christoffels.

Conclusion

Data preparation and load processes have a direct impact on how applications can begin and subsequently complete. Many organizations are using the Cloud platform to speed up the process, but to take full advantage of the infrastructure you have to apply the right software technology. SAS Viya enables the full realization of Cloud benefits through performance improvements, such as the transposing of data and the transformation of data using the DATA step or CAS Action Sets.

Additional Resources

SAS Global Forum Video: A Need For Speed: Loading Data via the Cloud
SAS Global Forum 2018 Paper: A Need For Speed: Loading Data via the Cloud
SAS Viya
SAS Viya Products


Read all the posts in this series.

Part 1: Technology that gets the most from the Cloud

Technology that gets the most from the Cloud was published on SAS Users.

5月 222018
 

Andy Dufresne, the wrongly convicted character in The Shawshank Redemption, provocatively asks the prison guard early in the film: “Do you trust your wife?” It’s a dead serious question regarding avoiding taxes on a recent financial windfall that had come the guard's way, and leads to events that eventually win [...]

AI and trust was published on SAS Voices by Leo Sadovy