SAS for Mixed Models: Introduction to Basic Applications takes you step-by-step through the journey of creating models, enabling you to transform scores of data into actionable insights.
The post The Model Gift – Transforming Data into Actionable Insights! appeared first on SAS Learning Post.
Phil Simon raises two key questions to ask in his second post on data literacy.
The post Questions on data literacy, part 2 appeared first on The Data Roundtable.
The REST architecture that SAS Viya is built on is, by its nature, open. This is a very powerful thing! In addition, the supplied command-line interfaces (CLIs) add a user-friendly interface to make it easier to make REST calls. Occasionally, however, it is necessary to call REST directly. This can occur when there is (currently) no CLI interface to a piece of functionality, or you wish to run a more complex task from a single command. In the SAS Global Enablement and Learning (GEL) group, as we staged our software images and developed our materials for our SAS Viya training, we found ourselves with some of these needs. As a result, we developed the GEL pyviyatools.
The GEL pyviyatools are a set of Python-based command-line tools that call the SAS Viya REST APIs. The tools can be used to make direct calls to any REST-endpoint (like a cURL command), and as a framework to build additional tools that make multiple rest calls to provide more complex functionality. The tools are designed to be used in conjunction with the sas-admin command line interfaces (CLI).
One of the challenges of making REST calls to SAS Viya is getting your authentication token. The tools simplify this issue by using the authentication mechanism provided by the SAS Viya command-line interfaces.
callrestapi (call_rest_api) is a general tool, and the building block for all the other tools. It calls a function callrestapi() that can also be used from any python program to build more complex tools.
The tools are self-documenting just like the Viya CLIs (just use the -h or –help option)
With callrestapi, you must pass a method and endpoint. You can optionally pass JSON data for a post request, content type headers, and the -o option to change the style of output.
In addition to this basic cURL-like functionality, there are some tools built on top of callrestapi that perform more complex functions. Here are few examples -- check out the GitHub project for a full list.
updatedomain.py loads a set of userids and passwords to a Viya domain from a csv file
listrules.py lists authorization rules subset on a principal and/or a uri
loginviauthinfo.py uses an authinfo file to authenticate to the CLI
updateprefences.py updates preferences for a user or group of users
updatedomain.py loads a set of userids and passwords to a SASViya domain from a csv file
createfolders.py creates a set of SAS Viya folders from a csv file
explainaccess.py explains access for a folder, object or service endpoint
You can get the tools from GitHub where the installation and usage instructions are documented
Please try these tools if you need more command-line functions in your SAS Viya environment. In addition, if you want to contribute additional tools built on the framework, please see the CONTRIBUTING.md file in the GitHub repository. You can also report any issues or suggestions via GitHub issues.
Introducing Python-based command-line tools for SAS Viya was published on SAS Users.
Many data analysts use a quantile-quantile plot (Q-Q plot) to graphically assess whether data can be modeled by a probability distribution such as the normal, lognormal, or gamma distribution. You can use the QQPLOT statement in PROC UNIVARIATE to create a Q-Q plot for about a dozen common distributions. However, it can be useful to use a variant of the Q-Q plot called the probability plot, which enables you to graphically assess how well a model fits the tails of the data. A probability plot can also be created in PROC UNIVARIATE. It is essentially a Q-Q plot in which the X axis is labeled nonlinearly by using percentiles of the model distribution. This article describes how to create and interpret a probability plot in SAS.
When fitting a distribution by using maximum likelihood estimation or some other method, you might notice that the model fits well in the center of the data but not in the tails. This may or may not be a problem. If you want the model your "typical" customers or patients, the model might be useful even if it does not fit perfectly in the tails.
Goodness-of-fit (GOF) tests indicate how well the model fits the data everywhere. Deviations in the tail can cause a GOF test to reject the hypothesis that the model fits well. You can use a probability plot to determine the percentiles at which the model begins to deviate from the data.
The following example is from the PROC UNIVARIATE documentation. The data are the thickness of the copper plating of 100 circuit boards. To illustrate that a model might fail to fit the tails of the data, I have artificially created four fake outliers and appended them to the end of the data. The example fits a normal distribution to the data and creates two Q-Q plots and a probability plot:
data Trans; input Thick @@; label Thick = 'Plating Thickness (mils)'; if _N_ <= 100 then Group="Real Data"; else Group = "Fake Data"; /* The last four observations are FAKE outliers */ datalines; 3.468 3.428 3.509 3.516 3.461 3.492 3.478 3.556 3.482 3.512 3.490 3.467 3.498 3.519 3.504 3.469 3.497 3.495 3.518 3.523 3.458 3.478 3.443 3.500 3.449 3.525 3.461 3.489 3.514 3.470 3.561 3.506 3.444 3.479 3.524 3.531 3.501 3.495 3.443 3.458 3.481 3.497 3.461 3.513 3.528 3.496 3.533 3.450 3.516 3.476 3.512 3.550 3.441 3.541 3.569 3.531 3.468 3.564 3.522 3.520 3.505 3.523 3.475 3.470 3.457 3.536 3.528 3.477 3.536 3.491 3.510 3.461 3.431 3.502 3.491 3.506 3.439 3.513 3.496 3.539 3.469 3.481 3.515 3.535 3.460 3.575 3.488 3.515 3.484 3.482 3.517 3.483 3.467 3.467 3.502 3.471 3.516 3.474 3.500 3.466 3.624 3.367 3.625 3.366 ; title 'Analysis of Plating Thickness'; proc univariate data=Trans noprint; qqplot Thick / normal grid odstitle="Q-Q Plot"; qqplot Thick / normal PCTLAXIS(grid) odstitle="Q-Q Plot with PCTLAXIS Option"; probplot Thick / normal grid odstitle="Probability Plot"; run; |
The first Q-Q plot indicates whether the model (in this case, the normal distribution) fits the data. If the bulk of the data falls along a straight line, then there is evidence that the model fits. In this case, the model fits most of the data but not the four (artificial) points that I added.
If these were all real data, you might wrestle with whether you should accept the normal model or choose an alternative model. The Q-Q plot indicates that the normal data seems to fit the central portion of the data.
It would be useful if the plot displayed the percentiles of the normal distribution. The next image shows the Q-Q plot with the PCTLAXES option (in the background) and a probability plot (in the foreground). Notice that the positions of the markers are the same as for the Q-Q plot; only the labels on the axes have changed. The PCTLAXES option for the second QQPLOT statement creates a graph that displays percentiles of the normal distribution at the top of the plot. The probability plot eliminates the quantiles altogether and displays only the normal percentiles
By using the probability plot, you can estimate that the model fits well for the central 95% of the data. Only the upper and lower 2.5th tails of the model do not fit the data. If your goal is to model only the central 95% of the data, it might be fine to ignore the extreme data.
I have previously shown how to construct a Q-Q plot from first principles. If you want to create a probability plot, you first create a Q-Q plot but then use the QUANTILE function to find the quantiles that correspond to the probabilities that you want to display on the axis. For example, the following SAS/IML statements print the quantiles for a set of typical probabilities:
proc iml; p = {0.001 0.01 0.05 0.10 0.25 0.5 0.75 0.9 0.95 0.99 0.999}; qntl = quantile("Normal", p); /* compute tick locations */ print (qntl // 100*p)[r={"Quantiles" "Percentiles"} F=Best4.]; quit; |
You can use the VALUES= and VALUESDISPLAY= options on the XAXIS statement in PROC SGPLOT to display the probability values at the locations of the corresponding quantiles. The following DATA step creates the coordinates for a Q-Q plot but uses the previous table of quantile values to specify the values and labels on the X axis. This can be useful, for example, if you want to customize the probability plot. For example, the following call sets the colors and symbols of the markers, adds a legend, and sets the labels for the axes.
proc sort data=Trans; by Thick; run; data ProbPlot; set Trans nobs=nobs; y = Thick; /* for convenience, call variable Y */ v = (_N_ - 0.375) / (nobs + 0.25); /* Blom (1958) */ q = quantile("Normal", v); run; title "Custom Probability Plot"; proc sgplot data=ProbPlot; scatter x=q y=y; yaxis grid label="Plating Thickness (mils)"; xaxis values=(-3.1 -2.3 -1.6 -1.3 -.67 0.00 0.67 1.28 1.64 2.33 3.09) valuesdisplay=('0.1' '1' '5' '10' '25' '50' '75' '90' '95' '99' '99.9') grid label="Normal Percentiles" fitpolicy=none; run; |
The VALUES= and VALUESDISPLAY= options are very useful. I use them regularly to customize the location of tick marks and the values that are displayed at each tick.
In summary, you can use the QQPLOT statement (with the PCTLAXIS option) or the PROBPLOT statement in PROC UNIVARIATE to create a probability plot. A probability plot is essentially the same as a Q-Q plot except that the X axis displays the percentiles of a model distribution instead of quantiles. If you want additional customization (or want to examine a model that is not supported by PROC UNIVARIATE), then you can create the Q-Q plot manually. You can use the VALUES= and VALUESDISPLAY= options on the XAXIS statement in PROC SGPLOT to display the percentiles of the model distribution. For more about how to interpret a probability plot (especially for a non-normal reference distribution), see the PROC UNIVARIATE documentation.
The post Create a probability plot in SAS appeared first on The DO Loop.
Singer-songwriter David Wilcox kept me out of prison. That’s not a country song gone bad – it’s a real-life experience. Almost two decades ago, I attended a David Wilcox concert in downtown Raleigh, North Carolina. His songs have always hooked my heart, and one in particular caught my attention that [...]
Rule number one: follow your heart was published on SAS Voices by Marcia Walker
View more Problem Solvers posts
Let’s begin with dates. We regularly hear customers say something similar to this: "I have a date, but I’m not sure how to use it or whether it’s even a SAS date yet." No worries--we can figure it out! A SAS date is a numeric variable whose value represents the number of days between January 1, 1960 and a specific date. For example, assume that you have a variable named X that has a value of 12398, but you’re not sure what that value represents. Is it a SAS date? Or does it represent January 23, 1998?
To determine what the value represents, you first need to run the CONTENTS procedure on the data set and determine whether the variable in question is character or numeric.
For this example, here is the partial output from the PROC CONTENTS step:
Alphabetic List of Variables and Attributes
# Variable Type Len Format
1 x Num 8
2 y Char 3
3 z Num 8 Z5.
data a;
mydate=12398;
format mydate worddate.;
run;
Obs mydate
1 December 11, 1993
data a;
x=12398;
y=input(put(x,5.),mmddyy6.);
format y date9.;
run;
Obs x y
1 12398 23JAN1998
data a;
x='12398';
y=input(x,mmddyy6.);
format y date9.;
run;
Many customers aren’t quite sure that they understand how to use arrays. Arrays are a common construct in many programming languages. Arrays can seem less complex when you remember that they are a temporary grouping of variables. When you perform the same operation on multiple variables, you have less to program if you can refer to a group of variables by a single name. You simply execute a DO loop that processes each variable in turn, and the task is complete!
We often see arrays used for "reshaping data" or transposing a data set from wide-to-long (or long-to-wide). For example, assume that you want to reshape a data set, comprised of three variables and four observations, into a data set that contains twelve variables. Using an array approach makes the programming much easier, as shown below:
Because the ALL array is populated by each observation as it is read from data set One, the END= option in the SET statement creates the variable LAST as a flag. This variable indicates when the last observation is read, and the IF statement tests variable LAST. If the variable has a value of 1 (which evaluates to "true"), the statement prints the contents of the program data vector to the output data set. Here's the starting data set and the reshaped result:
When I began working in Technical Support many years ago, the only interface to Base SAS® software was the Display Manager System, which has separate Program Editor, Log, and Output windows. Now, you can run SAS in various ways, and many of our customers use SAS Enterprise Guide and SAS Studio as their interfaces. One of the most frequently asked questions from customers is about how to access local PC files from these applications that access SAS through a UNIX server.
SAS Enterprise Guide offers built-in tasks to upload and download data sets and other files. You can find these tasks on the Tasks->Data menu.
Two of the tasks, Upload Data Files to Server and Download Data Files to PC, allow you to copy SAS data sets directly between your local PC and your SAS libraries. The third task, Copy Files, allow you to copy any file (or group of files) between your local PC and the file system of the SAS session. See this article to learn how to apply a common pattern with this task: export and download any file from SAS Enterprise Guide. (Note: The Copy Files task was added in SAS Enterprise Guide 7.13. For earlier releases, you can follow the steps in this article.)
If you’re using the SAS Studio interface, you can upload and download files between the server and your PC.
Upload File and Download File buttons in SAS Studio
To upload one or more files from your local computer:
The three topics that are discussed here don't represent new features or challenges. However, these topics generate many calls to Technical Support. It's a reminder that even as SAS continues to add new features and technology, we still need to know how to tackle the basic building blocks of our SAS programs.
FAQs about SAS dates, arrays and managing local PC files was published on SAS Users.
By now you’ve seen the headlines and the hype proclaiming data as the new oil. The well-meaning intent of these proclamations is to cast data in the role of primary economic driver for the 21st century, just as oil was for the 20th century. As analogies go, it’s not too [...]
No, data is not the new oil was published on SAS Voices by Leo Sadovy
Certain North Carolina counties have been in the news lately, for suspected election fraud involving absentee ballots in the 2018 election. Let's analyze the voter registration and absentee ballot data, to see if we can detect anything suspicious! In order to definitively determine whether fraud & illegal activity occurred, investigators [...]
The post Looking for indications of fraud, in North Carolina's absentee ballots appeared first on SAS Learning Post.
This article describes best practices and techniques that every data analyst should know before bootstrapping in SAS. The bootstrap method is a powerful statistical technique, but it can be a challenge to implement it efficiently. An inefficient bootstrap program can take hours to run, whereas a well-written program can give you an answer in an instant. If you prefer "instants" to "hours," this article is for you! I’ve compiled dozens of resources that explain how to compute bootstrap statistics in SAS.
Recall that a bootstrap analysis enables you to investigate the sampling variability of a statistic without making any distributional assumptions about the population. For example, if you compute the skewness of a univariate sample, you get an estimate for the skewness of the population. You might want to know the range of skewness values that you might observe from a second sample (of the same size) from the population. If the range is large, the original estimate is imprecise. If the range is small, the original estimate is precise. Bootstrapping enables you to estimate the range by using only the observed data.
In general, the basic bootstrap method consists of four steps:
The links in the previous list provide examples of best practices for bootstrapping in SAS. In particular, do not fall into the trap of using a macro loop to "resample, analyze, and append." You will eventually get the correct bootstrap estimates, but you might wait a long time to get them!
The remainder of this article is organized by the three ways to perform bootstrapping in SAS:
The articles in this section describe how to program the bootstrap method in SAS for basic univariate analyses, for regression analyses, and for related resampling techniques such as the jackknife and permutation tests. This section also links to articles that describe how to generate bootstrap samples in SAS.
When you bootstrap regression statistics, you have two choices for generating the bootstrap samples:
An important part of a bootstrapping is generating multiple bootstrap samples from the data. In SAS, there are many ways to obtain the bootstrap samples:
The SAS-supplied macros %BOOT, %JACK, and %BOOTCI, can perform basic bootstrap analyses and jackknife analyses. However, they require a familiarity with writing and using SAS macros. If you are interested, I wrote an example that shows how to use the %BOOT and %BOOTCI macros for bootstrapping. The documentation also provides several examples.
Many SAS procedures not only compute statistics but also provide standard errors or confidence intervals that enable you to infer whether an estimate is precise. Many confidence intervals are based on distributional assumptions about the population. ("If the errors are normally distributed, then....") However, the following SAS procedures provide an easy way to obtain a distribution-free confidence interval by using the bootstrap. See the SAS/STAT documentation for the syntax for each procedure.
Resampling techniques such as bootstrap methods and permutation tests are widely used by modern data analysts. But how you implement these techniques can make a huge difference between getting the results in a few seconds versus a few hours. This article summarizes and consolidates many previous articles that demonstrate how to perform an efficient bootstrap analysis in SAS. Bootstrapping enable you to investigate the sampling variability of a statistic without making any distributional assumptions. In particular, the bootstrap is often used to estimate standard errors and confidence intervals for parameters.
The post The essential guide to bootstrapping in SAS appeared first on The DO Loop.
In her role as Product Manager for SAS Platform Technologies (including the SAS Add-In for Microsoft Office), my colleague Amy Peters hears this question often. With many organizations adopting Microsoft Office 365 -- the "cloud" version of Office -- what does this mean for other processes that integrate with Microsoft Office applications?
Microsoft has used different names for these similar offerings: Office 2016, Office 365, Microsoft 365, Office Online. The bottom line is that most users of a "365" package in the cloud, also have access to the Microsoft Office tools on their Windows desktop. They can use the full version of Excel, PowerPoint, Word, etc., and they also have access to these same tools via a web browser. At SAS, we recently experienced this transition ourselves. Have the Office applications on our desktops vanished? No, they have not. While more of our data is now on the cloud (looking at you, OneDrive), it's not really changing how we work, especially when creating/maintaining content. (Like many organizations, we already had one foot in this world by using Microsoft SharePoint for collaboration.)
Let's look at an example of how I use SAS with Microsoft Office. First, I create a report in SAS Visual Analytics. Then I open Excel on my desktop and use the SAS Add-In for Microsoft Office to embed the shared report into my spreadsheet. Want to see what that looks like in action? Check out this video Tech Talk with SAS developer Tim Beese.
Now suppose that I share this content in Microsoft OneDrive, and my colleague views it in Excel in a web browser. Yep, the content is still there. The difference is that the content is not dynamic like it is on my desktop. So what do you do when you want to edit that spreadsheet displaying in the browser? You select Open in Excel and the document opens on your desktop. Voila! The content is dynamic and you have all the functionality the SAS Add-In for Microsoft Office provides.
Today, the expectation of most users working with "Office Online" applications in their browsers is that it's primarily for viewing and basic editing. Will this change? Probably. We're researching how to provide more of the SAS Add-In for Microsoft Office function in a browser app. If you or your colleagues need this browser-based function – you want to do something specific in Excel with your SAS content -- we want to hear from you. And do you have a plan to move completely to browser-based Office apps? Currently you can't create SAS content from a browser-based Office app. If that's a pressing need, we would like to know. For now, we're not hearing of use cases where some form of the desktop app isn't still in the picture.
SAS integration with these everyday productivity tools, like Microsoft Office, is important to us. Don't forget about these SAS programming methods to create and read your Microsoft Office content:
How are you using Microsoft Office 365 with SAS? How do you think this workflow will change for you in the next year or two? Leave a comment -- we would love to hear from you.
The post Does SAS support Microsoft Office 365? appeared first on The SAS Dummy.