8月 052020
 

Have you ever seen the "brain teaser" for children that shows a 4 x 4 grid and asks "how many squares of any size are in this grid?" To solve this problem, the reader must recognize that there are sixteen 1 x 1 squares, nine 2 x 2 squares, four 3 x 3 squares, and one 4 x 4 square. Hence there are a total of (16+9+4+1) = 30 squares of any size.

Recently a SAS statistical programmer asked a similar question about submatrices of a matrix. Specifically, the programmer wanted to form square submatrices of consecutive rows and columns. This article shows a few tricks in the SAS/IML language that enable you to generate and compute with all k x k submatrices of a matrix, where the submatrices are formed by using consecutive rows and columns of the original matrix. Although it is possible to consider more general submatrices, in this article, a "submatrix" always refers to one that is formed by using consecutive rows and columns.

Use subscripts to form submatrices

I've previously written about several ways to extract submatrices from a matrix. Recall the "colon operator" (formally called the index creation operator) generates a vector of consecutive integers. You can use the vectors in the row and column position of the subscripts to extract a submatrix. For example, the following SAS/IML statements define a 4 x 4 matrix and extract the four 3 x 3 submatrices:

A = {4 3 1 6,
     2 4 3 1,
     0 2 4 3,
     5 0 2 4};
/* extract the four 3 x 3 submatrices */
A11 = A[1:3, 1:3];
A12 = A[1:3, 2:4];
A21 = A[2:4, 1:3];
A22 = A[2:4, 2:4];

Manually specifying the row and column numbers is tedious. Let's write a SAS/IML function that will return a k x k submatrix of A by starting at the (i,j)th position of A. Call k the order of the submatrix.

/* Given A, return the square submatrix of order k starting at index (i,j) */
start submat(A, i, j, k);
   return A[ i:(i+k-1), j:(j+k-1) ];
finish;
 
/* example : 3 x 3 submatrices of A */
A11 = submat(A, 1, 1, 3);
A12 = submat(A, 1, 2, 3);
A21 = submat(A, 2, 1, 3);
A22 = submat(A, 2, 2, 3);
 
/* https://blogs.sas.com/content/iml/2015/12/02/matrices-graphs-gridded-layout.html */
ods layout gridded columns=4 advance=table;
print A11, A12, A21, A22;
ods layout end;

The number of submatrices

If A is an n x m matrix, we can ask how many submatrices of order k are in A. Recall that we are only considering submatrices formed by consecutive rows and columns. There are (n – k + 1) sequences of consecutive rows of length k, such as 1:k, 2:(k+1), and so forth. Similarly, there are (m – k + 1) sequences of consecutive columns of length k. So the following SAS/IML function counts the number of submatrices of order k. You can call the function for different k=1, 2, 3, and 4 in order to solve the "Counting Squares" brain teaser:

/* count all submatrices of order k for the matrix A */
start CountAllSubmat(A, k);
   numRows = nrow(A)-k+1;
   numCols = ncol(A)-k+1;
   return numRows * numCols;          
finish;
 
n = nrow(A);
numSubmat = j(n, 1, .);
do k = 1 to n;
   numSubmat[k] = CountAllSubmat(A, k);
end;
order = T(1:n);
print order numSubmat;

If you add up the number of squares each size, you get a total of 30 squares.

Iterate over submatrices of a specified order

You can iterate over all submatrices of a specified order. You can perform a matrix computation on each submatrix (for example, compute its determinant), or you can pack it into a list for future processing. The following SAS/IML function iterates over submatrices and puts each submatrix into a list. The STRUCT subroutine from the ListUtil package can be used to indicate the contents of the list in a concise form.

start GetListSubmat(A, k);
   numSub = CountAllSubmat(A, k);
   L = ListCreate( numSub );           /* create list with numSub items */
   cnt = 1;
   do i = 1 to nrow(A)-k+1;            /* for each row */
      do j = 1 to ncol(A)-k+1;         /* and for each column */
         B = submat(A, i, j, k);       /* compute submatrix of order k from (i,j)th cell */
         call ListSetItem(L, cnt, B);  /* put in list (or compute with B here) */
         cnt = cnt + 1;
      end;
   end;
   return L;
finish;
 
/* Example: get all matrices of order k=2 */
L = GetListSubmat(A, 2);
 
package load ListUtil;
call Struct(L);

The output from the STRUCT call shwos that there are nine 2 x 2 matrices. The Value1–Valuie4 columns show the values (in row-major order). For example, the first 2 x 2 submatrix (in the upper left corner of A) is {4 3, 2 4}. The second submatrix (beginning with A[1,2]) is {3 1, 4 3}.

It is straightforward to modify the function to compute the determinant or some other matrix computation.

Summary

This article shows some tips and techniques for dealing with submatrices of a matrix. The submatrices in this article are formed by using consecutive rows and columns. Thus, they are similar to the "Counting Squares" brain teaser, which requires that you consider sub-squares of order 1, 2, 3, and so forth.

The post Submatrices of matrices appeared first on The DO Loop.

8月 032020
 

Help us celebrate our collective success! Our SAS Support Communities are nominated for Best-in-Class: Community and Bottom Line Savings Rock Star. Voting closes August 14.

Voting has opened for the first annual Khoros Kudos Awards, and this year the SAS Communities has been nominated for two categories:

Join us in celebrating the growth and success of our online communities by casting your vote by 5 pm PT on Friday, August 14! To vote, click the category name above, scroll to the end of the entry and click ‘Kudo’. (Note: Voting requires registering, but it’s quick and Khoros won’t spam you).

The SAS Support Communities are home to nearly 340k members

Not familiar with the SAS Support Communities? Our Communities are a wonderful resource for SAS users of all levels – and a great way to connect, share code and explore with fellow SAS aficionados, academics, learners, professionals, experts – and employees, too!

SAS Support Communities began more than 15 years ago, and have grown to nearly 340k members (22,800 new members this year alone!) and are home to over 600k discussions around topics like SAS products, solutions, administration and training. The SAS Support Communities are designed to help you succeed, and they’re a great place to network, collaborate and uncover insights with fellow SAS users. Members can take advantage of specially curated content, peer and expert recommendations, critical product updates and more.

Need help? Ask the community any time of day!  Start a conversation to get tips and share what you know with others.  Our team of SAS experts monitor the forums 24/7 to provide assistance when you need it most – and maintains a community library of user-friendly resources like how-to tutorials and handy tips. If you’re new to the SAS Communities, check out our New SAS User community and our welcome resources to get started!

Thank you for being a part of our SAS Support Communities and Happy Posting!

Cast your vote! SAS Support Communities are on the 2020 Khoros Kudos Awards Ballot! was published on SAS Users.

8月 032020
 

The iml action was introduced in Viya 3.5. As shown in a previous article, the iml action supports ways to implement the map-reduce paradigm, which is a way to distribute a computation by using multiple threads. The map-reduce paradigm is ideal for “embarrassingly parallel” computations, which are composed of many independent and essentially identical computations. I wrote an example program that shows how to run a Monte Carlo simulation in parallel by using the iml action in SAS Viya.

The map-reduce paradigm is often used when there is ONE task to perform and you want to assign a portion of the work to each thread. A different scenario is when you have SEVERAL independent tasks to perform. You can assign each task to a separate thread and ask each thread to compute an entire task. The iml action supports the PARTASKS function (short for "parallel tasks") for running tasks in separate threads. This article shows a simple example.

Three tasks run sequentially

Suppose you have three tasks. The first takes 2 seconds to run, the next takes 0.5 seconds, and the third takes 1 second. If you run a set of tasks sequentially, the total time is the sum of the times for the individual tasks: 2 + 0.5 + 1 = 3.5 seconds. In contrast, if you run the tasks in parallel, the total time is the maximum of the individual times: 2 seconds. Running the tasks in parallel saves time, and the speed-up depends on the ratio of the largest time to the total time, which is 2/3.5 in this example. Thus, the time for the parallel computation is 57% of the time for the sequential computation.

In general, the largest speedup occurs when the tasks each run in about the same time. In that situation (ignoring overhead costs), the time reduction can be as much 1/k when you run k tasks in k threads.

Let's run three tasks sequentially, then run the same tasks in parallel. Suppose you have a large symmetric N x N matrix and you need to compute three things: the determinant, the matrix inverse, and the eigenvalues. The following call to PROC IML runs the three tasks sequentially. The TOEPLITZ function is used to create a large symmetric matrix.

proc iml;
N = 2000;
A = toeplitz( (N:1)/100 );     /* N x N symmetric matrix */
 
det = det(A);                  /* get det(A) */
inv = inv(A);                  /* get inv(A) */
eigval = eigval(A);            /* get eigval(A) */

The previous computations are performed by directly calling three built-in SAS/IML functions. Of course, in practice, the tasks will be more complex than these. Almost surely, each task will be encapsulated into a SAS/IML module. To emulate that process, I am going to define three trivial "task modules." The following statements perform The same computations, but now each "task" is performed by calling a user-written module:

start detTask(A);
   return det(A);
finish;
start invTask(A);
   return inv(A);
finish;
start eigvalTask(A);
   return eigval(A);
finish;
 
det = detTask(A);              /* get det(A) */
inv = invTask(A);              /* get inv(A) */
eigval = eigvalTask(A);        /* get eigval(A) */

The syntax of the PARTASKS function

You can use the PARTASKS function to run the three tasks concurrently. To use the PARTASKS function, each task must be a user-defined SAS/IML function that has exactly one input argument and returns one output argument. For our example, the detTask, invTask, and eigvalTask modules satisfy these requirements. (If a task requires more than one input argument, you can pack the arguments into a list and pass the list as a single argument.)

The syntax of the PARTASKS function is
result = PARTASKS( Tasks, TaskArgs, Options );
where

  • Tasks is a character vector that names the SAS/IML functions. For our example, Tasks = {'detTask' 'invTask' 'eigvalTask'};
  • TaskArgs is a list of arguments. The i_th argument will be sent to the i_th module. In this example, each module takes the same matrix, so you can create a list that has three items, each being the same matrix: TaskArgs = [A, A, A];
  • Options is an (optional) vector that specifies how the tasks are distributed. The documentation of the PARTASKS function gives the full details, but the first element of the vector specifies how to distribute the computation to nodes and threads, and the second element specifies whether to print information about the performance of the tasks. For this example, you can choose options = {1, 1};, which will distribute tasks to threads on the controller node and print information about the tasks.

Parallel tasks

Suppose you are running SAS Viya and you use the have access to a machine that has at least four threads. You can use the CAS statement to define a session that will use only one machine (no worker nodes). The CAS statement might look something like this:
cas sess0 host='your_host_name' casuser='your_username' port=&YourPort; /* SMP, 0 workers */
Then you can run the PARTASKS function by making the following call to the iml action. If you are not familiar with the iml action, see the getting started example for the iml action.

proc cas;
session sess0;                         /* SMP session: controller node only    */
loadactionset 'iml';                   /* load the action set               */
source partasks;                       /* put program in SOURCE/ENDSOURCE block */
   start detTask(A);                   /* 1st task */
      return det(A);
   finish;
   start invTask(A);                   /* 2nd task */
      return inv(A);
   finish;
   start eigvalTask(A);                /* 3rd task */
      return eigval(A);
   finish;
 
   /* ----- Main Program ----- */
   N = 2000;
   A = toeplitz( (N:1)/100 );          /* N x N symmetric matrix */
   Tasks = {'detTask' 'invTask' 'eigvalTask'};
   Args = [A, A, A];                   /* each task gets same arg in this case */
   opt = {1,                           /* distribute to threads on controller  */
          1};                          /* display information about the tasks  */
   Results = ParTasks(Tasks, Args, opt); /* results are returned in a list     */
 
   /* the i_th list item is the result of the i_th task */
   det    = Results$1;                 /* get det(A)                           */
   inv    = Results$2;                 /* get inv(A)                           */
   eigval = Results$3;                 /* get eigval(A)                        */
endsource;
iml / code=partasks nthreads=4;
run;

The results are identical to the sequential computations in PROC IML. However, this program executes the three tasks in parallel, so the total time for the computations is the time required by the longest task.

Visualize the parallel tasks computations

You can use the following diagram to help visualize the program. (Click to enlarge.)

The diagram shows the following:

  • The CAS session does not include any worker nodes. Therefore, the iml action runs entirely on the controller, although it can still use multiple threads. This mode is known as symmetric multiprocessing (SMP) or single-machine mode. Notice that the call to the iml action (the statement just before the RUN statement) specifies the NTHREADS=4 parameter, which causes the action to use four threads. Because there are only three tasks, one thread remains idle.
  • The program defines the matrix, A, the vector of module names, and the list of arguments. When you call the ParTasks function, the i_th argument is sent to the i_th module on the i_th thread.
  • Each thread runs a different function module. Each function returns its result. The results are returned in a list. The i_th item in the list is the result of the call to the i_th function module.

Summary

This article demonstrates a simple call to the PARTASKS function in the iml action, which is available in SAS Viya 3.5. The example shows how to distribute tasks among k threads. Each thread runs concurrently and independently. The total time for the computation depends on the longest time that any one thread spends on its computation.

This example is simple but demonstrates the main ideas of distributing tasks in parallel. In a future article, I will present a more compelling example.

For more information and examples, see Wicklin and Banadaki (2020), "Write Custom Parallel Programs by Using the iml Action," which is the basis for these blog posts. Another source is the SAS IML Programming Guide, which includes documentation and examples for the iml action.

The post Run tasks in parallel in SAS Viya appeared first on The DO Loop.

7月 312020
 

In this SAS demo, I demonstrate how to prompt for a date range in a SAS Visual Analytics report. We walk through how to configure prompts using four types of control objects:

Slider (range)

Slider (single value)

Drop-down list

Text input

Then we cover how to define parameters and configure filters for report objects using the date prompted boundaries. These techniques are demonstrated using SAS Viya 3.4 SAS Visual Analytics 8.4 but can be applied to earlier releases.

I've also written accompanying articles in a four-part series. Below the video see the links to the articles and their corresponding timestamps in the video.

01:33 – 1st Example: Slider (Range) - How to prompt for a date range in a SAS VA report – Example 1 Slider (Range)

03:01 – 2nd Example: Slider (Single Value) - How to prompt for a date range in a SAS VA report – Example 2 Slider (Single Value)

14:04 – 3rd Example: Drop-down list - How to prompt for a date range in a SAS VA report – Example 3 Drop-down list

19:33 – 4th Example: Text input - How to prompt for a date range in a SAS VA report – Example 4 Text input

Other References

How to prompt for a date range in a SAS Visual Analytics Report - Four Part Series was published on SAS Users.

7月 302020
 
How often have you needed Google Translate, but for SAS code?

SAS Technical Support often gets requests like the following: "I have this API named <insert name of really cool API here> and I want to process data I get back from the API with SAS. How do I do it?"

In this article, I'll show you how to translate the examples from your API documentation (offered in a select few languages) to the equivalent in SAS.

Test your API with cURL

My first recommendation is to know your API. Most APIs come with documentation and will give specific guidance regarding how to POST data (send) to the API and how to GET data (receive) from the API. Often, examples are provided using cURL commands.

With this information, you are welcome to examine the SAS documentation for the HTTP procedure and build your code. Before you call or email SAS Technical Support asking for PROC HTTP code, I encourage you to verify that you can communicate with your API (or URL) from outside of SAS. One way to do so is with cURL. cURL (Client URL) is a command-line tool that is shipped with many UNIX flavors and installed easily on Windows. With a cURL command, you can interact with your URLs from outside of SAS from a command-line prompt like this:

curl -o "c:\temp\file.txt" -request "https://httpbin.org/get"

From SAS, you can use a cURL command with the DATA step, like this:

data _null_;
 %sysexec curl -o "c:\temp\file.txt" -request "https://httpbin.org/get";
run;

File.txt contains the response from the URL:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "Range": "bytes=Request", 
    "User-Agent": "curl/7.46.0", 
    "X-Amzn-Trace-Id": "Root=1-5f028cd3-2ec7e1e05da1f616e9106ee8"
  }, 
  "origin": "149.173.8.1", 
  "url": "https://httpbin.org/get"
}

However, if you use SAS® Enterprise Guide® or SAS® Studio, you might not have permissions to run operating system commands from SAS, so you need a way to translate your cURL commands to SAS. The previous cURL command is easily translated to the following PROC HTTP code:

filename out "c:\temp\file.txt";
proc http url="https://httpbin.org/get" out=out;
run;
  1. The -o (OUTPUT) cURL argument translates to the OUT= argument in PROC HTTP.
  2. The -request argument defaults to a GET for cURL (also the default for PROC HTTP, so METHOD=“GET” is the correct syntax but unnecessary for this step).
  3. Note: The URL= argument is always quoted.

The cURL command supports many options and features. Check out the cURL reference page. SAS can't guarantee that all are directly translatable to PROC HTTP, but I do want to cover some of the most popular that SAS customers have asked about.

Sending data to an API

If your cURL command uses the -d (DATA) option, you'll use the IN= argument in your PROC HTTP statement. Here I am posting to the URL httpbin.org/post a file called test.csv, which resides in my c:\temp directory:

curl -d "c:\temp\test.csv" -X post "https://httpbin.org/post";

This command translates to the following PROC HTTP code:

filename test "c:\temp\test.csv";
 
proc http url="https://httpbin.org/post"
 /* If the IN= argument is used then method="post" is the default */ 
 /* and therefore unnecessary in this step */
 method="post"
 in=test;
run;

Working with authentication

None of the URLs above require authentication, but you'll likely find authentication is part of most APIs. Many APIs have moved to OAuth for authentication. This method of authentication requires the use of an access token, which you obtain with a POST command. With the correct credentials, this cURL command posts to the SAS® Viya® SASLogon REST API in order to obtain an access token:

 
curl -X POST "https://server.example.com/SASLogon/oauth/token" \
      -H "Content-Type: application/x-www-form-urlencoded" \
      -d "grant_type=password&username=userid&password=mypassword" \
      -u "app:mysecret" -o "c:\temp\token.txt"

The following PROC HTTP code does the same task:

 filename out temp;
 proc http url="http://servername/SASLogon/oauth/token"
    in="grant_type=password&username=userid&password=mypassword"
    webusername="clientid"
    webpassword="clientsecret"
    method="post"
    out=out;
   headers "Content-Type"="application/x-www-form-urlencoded";
run;
  1. The -u option is for user ID and password.*
  2. The -o command/output captures the response, in this case a JSON file. In this case you mimic -o with a FILENAME statement to write the text in JSON format to the WORK library location.
  3. The -H command is popular and translates to the HEADERS statement in PROC HTTP.

*Read about ways to hide your credentials in Chris Hemedinger's post here: How to secure your REST API credentials in SAS programs.

The output file contains an access token, necessary to make requests on behalf of a client to the REST API. In this example, a cURL command like the following requests a list of folders from the Folders microservice:

curl -X GET "https://server.example.com/folders/folders/@myFolder" \
      -H "Accept: application/json" \
      -H "Authorization: Bearer TOKEN-STRING"

The PROC HTTP code will look like this if you "directly translate":

filename new temp;
 
proc http url=" https://server.example.com/folders/folders/@myFolder"
 method="get" out=new;
 headers "Accept"="application/json" 
         "Authorization"="Bearer TOKEN-STRING";
run;

But starting in SAS® 9.4M5, there's a shortcut with the OAUTH_BEARER option:

filename new temp;
 
proc http OAUTH_BEARER="TOKEN-STRING" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Processing JSON responses with the JSON engine

I can't tell you about PROC HTTP without a mention of the JSON engine. Starting in SAS® 9.4m4, the JSON engine enables us to easily read JSON files. I can use the previous cURL command to pipe my access token to a file with the -o argument, but using my PROC HTTP code I can easily move that value into a macro variable. I'll add a LIBNAME statement that points to the fileref in the previous step:

libname test json fileref=new;

I can then examine the contents of the JSON output with this step:

proc contents data=test._all_;
run;

Here I spy the access token I will need for a later PROC HTTP step:

Here's how I can place it in a macro variable:

data _null_;
 set test.root;
 call symputx("access_token",access_token);
run;
 
%put &access_token;

So everywhere I used TOKEN-STRING in the previous code, I can now use the macro variable instead, like this:

proc http OAUTH_BEARER="&access_token" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Debugging with PROC HTTP

With the cURL command, you can use the -v (verbose) argument to get connection and header information. It's helpful for debugging and diagnosing trouble spots.

In SAS® 9.4M5, the DEBUG statement was added to PROC HTTP. DEBUG supports several options. I'll highlight the LEVEL= option here:

 Level= 0 | 1 | 2 | 3

Selecting 0 provides no debugging information while 3 provides the highest amount of data and messages. Base SAS 9.4 Procedures Guide includes full descriptions of each debug level.

See the HTTP Procedure documentation for additional syntax.

Generating cURL commands with Postman

If you want someone to "write your code for you," I recommend using a product like Postman to test your POST and GET commands with your API from outside of SAS. Postman is an open-source product with a super cool feature: it will produce CURL commands from successful communication with a URL.

Recommended resources

How to translate your cURL command into SAS code was published on SAS Users.

7月 302020
 
How often have you needed Google Translate, but for SAS code?

SAS Technical Support often gets requests like the following: "I have this API named <insert name of really cool API here> and I want to process data I get back from the API with SAS. How do I do it?"

In this article, I'll show you how to translate the examples from your API documentation (offered in a select few languages) to the equivalent in SAS.

Test your API with cURL

My first recommendation is to know your API. Most APIs come with documentation and will give specific guidance regarding how to POST data (send) to the API and how to GET data (receive) from the API. Often, examples are provided using cURL commands.

With this information, you are welcome to examine the SAS documentation for the HTTP procedure and build your code. Before you call or email SAS Technical Support asking for PROC HTTP code, I encourage you to verify that you can communicate with your API (or URL) from outside of SAS. One way to do so is with cURL. cURL (Client URL) is a command-line tool that is shipped with many UNIX flavors and installed easily on Windows. With a cURL command, you can interact with your URLs from outside of SAS from a command-line prompt like this:

curl -o "c:\temp\file.txt" -request "https://httpbin.org/get"

From SAS, you can use a cURL command with the DATA step, like this:

data _null_;
 %sysexec curl -o "c:\temp\file.txt" -request "https://httpbin.org/get";
run;

File.txt contains the response from the URL:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "Range": "bytes=Request", 
    "User-Agent": "curl/7.46.0", 
    "X-Amzn-Trace-Id": "Root=1-5f028cd3-2ec7e1e05da1f616e9106ee8"
  }, 
  "origin": "149.173.8.1", 
  "url": "https://httpbin.org/get"
}

However, if you use SAS® Enterprise Guide® or SAS® Studio, you might not have permissions to run operating system commands from SAS, so you need a way to translate your cURL commands to SAS. The previous cURL command is easily translated to the following PROC HTTP code:

filename out "c:\temp\file.txt";
proc http url="https://httpbin.org/get" out=out;
run;
  1. The -o (OUTPUT) cURL argument translates to the OUT= argument in PROC HTTP.
  2. The -request argument defaults to a GET for cURL (also the default for PROC HTTP, so METHOD=“GET” is the correct syntax but unnecessary for this step).
  3. Note: The URL= argument is always quoted.

The cURL command supports many options and features. Check out the cURL reference page. SAS can't guarantee that all are directly translatable to PROC HTTP, but I do want to cover some of the most popular that SAS customers have asked about.

Sending data to an API

If your cURL command uses the -d (DATA) option, you'll use the IN= argument in your PROC HTTP statement. Here I am posting to the URL httpbin.org/post a file called test.csv, which resides in my c:\temp directory:

curl -d "c:\temp\test.csv" -X post "https://httpbin.org/post";

This command translates to the following PROC HTTP code:

filename test "c:\temp\test.csv";
 
proc http url="https://httpbin.org/post"
 /* If the IN= argument is used then method="post" is the default */ 
 /* and therefore unnecessary in this step */
 method="post"
 in=test;
run;

Working with authentication

None of the URLs above require authentication, but you'll likely find authentication is part of most APIs. Many APIs have moved to OAuth for authentication. This method of authentication requires the use of an access token, which you obtain with a POST command. With the correct credentials, this cURL command posts to the SAS® Viya® SASLogon REST API in order to obtain an access token:

 
curl -X POST "https://server.example.com/SASLogon/oauth/token" \
      -H "Content-Type: application/x-www-form-urlencoded" \
      -d "grant_type=password&username=userid&password=mypassword" \
      -u "app:mysecret" -o "c:\temp\token.txt"

The following PROC HTTP code does the same task:

 filename out temp;
 proc http url="http://servername/SASLogon/oauth/token"
    in="grant_type=password&username=userid&password=mypassword"
    webusername="clientid"
    webpassword="clientsecret"
    method="post"
    out=out;
   headers "Content-Type"="application/x-www-form-urlencoded";
run;
  1. The -u option is for user ID and password.*
  2. The -o command/output captures the response, in this case a JSON file. In this case you mimic -o with a FILENAME statement to write the text in JSON format to the WORK library location.
  3. The -H command is popular and translates to the HEADERS statement in PROC HTTP.

*Read about ways to hide your credentials in Chris Hemedinger's post here: How to secure your REST API credentials in SAS programs.

The output file contains an access token, necessary to make requests on behalf of a client to the REST API. In this example, a cURL command like the following requests a list of folders from the Folders microservice:

curl -X GET "https://server.example.com/folders/folders/@myFolder" \
      -H "Accept: application/json" \
      -H "Authorization: Bearer TOKEN-STRING"

The PROC HTTP code will look like this if you "directly translate":

filename new temp;
 
proc http url=" https://server.example.com/folders/folders/@myFolder"
 method="get" out=new;
 headers "Accept"="application/json" 
         "Authorization"="Bearer TOKEN-STRING";
run;

But starting in SAS® 9.4M5, there's a shortcut with the OAUTH_BEARER option:

filename new temp;
 
proc http OAUTH_BEARER="TOKEN-STRING" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Processing JSON responses with the JSON engine

I can't tell you about PROC HTTP without a mention of the JSON engine. Starting in SAS® 9.4m4, the JSON engine enables us to easily read JSON files. I can use the previous cURL command to pipe my access token to a file with the -o argument, but using my PROC HTTP code I can easily move that value into a macro variable. I'll add a LIBNAME statement that points to the fileref in the previous step:

libname test json fileref=new;

I can then examine the contents of the JSON output with this step:

proc contents data=test._all_;
run;

Here I spy the access token I will need for a later PROC HTTP step:

Here's how I can place it in a macro variable:

data _null_;
 set test.root;
 call symputx("access_token",access_token);
run;
 
%put &access_token;

So everywhere I used TOKEN-STRING in the previous code, I can now use the macro variable instead, like this:

proc http OAUTH_BEARER="&access_token" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Debugging with PROC HTTP

With the cURL command, you can use the -v (verbose) argument to get connection and header information. It's helpful for debugging and diagnosing trouble spots.

In SAS® 9.4M5, the DEBUG statement was added to PROC HTTP. DEBUG supports several options. I'll highlight the LEVEL= option here:

 Level= 0 | 1 | 2 | 3

Selecting 0 provides no debugging information while 3 provides the highest amount of data and messages. Base SAS 9.4 Procedures Guide includes full descriptions of each debug level.

See the HTTP Procedure documentation for additional syntax.

Generating cURL commands with Postman

If you want someone to "write your code for you," I recommend using a product like Postman to test your POST and GET commands with your API from outside of SAS. Postman is an open-source product with a super cool feature: it will produce CURL commands from successful communication with a URL.

Recommended resources

How to translate your cURL command into SAS code was published on SAS Users.

7月 302020
 
How often have you needed Google Translate, but for SAS code?

SAS Technical Support often gets requests like the following: "I have this API named <insert name of really cool API here> and I want to process data I get back from the API with SAS. How do I do it?"

In this article, I'll show you how to translate the examples from your API documentation (offered in a select few languages) to the equivalent in SAS.

Test your API with cURL

My first recommendation is to know your API. Most APIs come with documentation and will give specific guidance regarding how to POST data (send) to the API and how to GET data (receive) from the API. Often, examples are provided using cURL commands.

With this information, you are welcome to examine the SAS documentation for the HTTP procedure and build your code. Before you call or email SAS Technical Support asking for PROC HTTP code, I encourage you to verify that you can communicate with your API (or URL) from outside of SAS. One way to do so is with cURL. cURL (Client URL) is a command-line tool that is shipped with many UNIX flavors and installed easily on Windows. With a cURL command, you can interact with your URLs from outside of SAS from a command-line prompt like this:

curl -o "c:\temp\file.txt" -request "https://httpbin.org/get"

From SAS, you can use a cURL command with the DATA step, like this:

data _null_;
 %sysexec curl -o "c:\temp\file.txt" -request "https://httpbin.org/get";
run;

File.txt contains the response from the URL:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "Range": "bytes=Request", 
    "User-Agent": "curl/7.46.0", 
    "X-Amzn-Trace-Id": "Root=1-5f028cd3-2ec7e1e05da1f616e9106ee8"
  }, 
  "origin": "149.173.8.1", 
  "url": "https://httpbin.org/get"
}

However, if you use SAS® Enterprise Guide® or SAS® Studio, you might not have permissions to run operating system commands from SAS, so you need a way to translate your cURL commands to SAS. The previous cURL command is easily translated to the following PROC HTTP code:

filename out "c:\temp\file.txt";
proc http url="https://httpbin.org/get" out=out;
run;
  1. The -o (OUTPUT) cURL argument translates to the OUT= argument in PROC HTTP.
  2. The -request argument defaults to a GET for cURL (also the default for PROC HTTP, so METHOD=“GET” is the correct syntax but unnecessary for this step).
  3. Note: The URL= argument is always quoted.

The cURL command supports many options and features. Check out the cURL reference page. SAS can't guarantee that all are directly translatable to PROC HTTP, but I do want to cover some of the most popular that SAS customers have asked about.

Sending data to an API

If your cURL command uses the -d (DATA) option, you'll use the IN= argument in your PROC HTTP statement. Here I am posting to the URL httpbin.org/post a file called test.csv, which resides in my c:\temp directory:

curl -d "c:\temp\test.csv" -X post "https://httpbin.org/post";

This command translates to the following PROC HTTP code:

filename test "c:\temp\test.csv";
 
proc http url="https://httpbin.org/post"
 /* If the IN= argument is used then method="post" is the default */ 
 /* and therefore unnecessary in this step */
 method="post"
 in=test;
run;

Working with authentication

None of the URLs above require authentication, but you'll likely find authentication is part of most APIs. Many APIs have moved to OAuth for authentication. This method of authentication requires the use of an access token, which you obtain with a POST command. With the correct credentials, this cURL command posts to the SAS® Viya® SASLogon REST API in order to obtain an access token:

 
curl -X POST "https://server.example.com/SASLogon/oauth/token" \
      -H "Content-Type: application/x-www-form-urlencoded" \
      -d "grant_type=password&username=userid&password=mypassword" \
      -u "app:mysecret" -o "c:\temp\token.txt"

The following PROC HTTP code does the same task:

 filename out temp;
 proc http url="http://servername/SASLogon/oauth/token"
    in="grant_type=password&username=userid&password=mypassword"
    webusername="clientid"
    webpassword="clientsecret"
    method="post"
    out=out;
   headers "Content-Type"="application/x-www-form-urlencoded";
run;
  1. The -u option is for user ID and password.*
  2. The -o command/output captures the response, in this case a JSON file. In this case you mimic -o with a FILENAME statement to write the text in JSON format to the WORK library location.
  3. The -H command is popular and translates to the HEADERS statement in PROC HTTP.

*Read about ways to hide your credentials in Chris Hemedinger's post here: How to secure your REST API credentials in SAS programs.

The output file contains an access token, necessary to make requests on behalf of a client to the REST API. In this example, a cURL command like the following requests a list of folders from the Folders microservice:

curl -X GET "https://server.example.com/folders/folders/@myFolder" \
      -H "Accept: application/json" \
      -H "Authorization: Bearer TOKEN-STRING"

The PROC HTTP code will look like this if you "directly translate":

filename new temp;
 
proc http url=" https://server.example.com/folders/folders/@myFolder"
 method="get" out=new;
 headers "Accept"="application/json" 
         "Authorization"="Bearer TOKEN-STRING";
run;

But starting in SAS® 9.4M5, there's a shortcut with the OAUTH_BEARER option:

filename new temp;
 
proc http OAUTH_BEARER="TOKEN-STRING" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Processing JSON responses with the JSON engine

I can't tell you about PROC HTTP without a mention of the JSON engine. Starting in SAS® 9.4m4, the JSON engine enables us to easily read JSON files. I can use the previous cURL command to pipe my access token to a file with the -o argument, but using my PROC HTTP code I can easily move that value into a macro variable. I'll add a LIBNAME statement that points to the fileref in the previous step:

libname test json fileref=new;

I can then examine the contents of the JSON output with this step:

proc contents data=test._all_;
run;

Here I spy the access token I will need for a later PROC HTTP step:

Here's how I can place it in a macro variable:

data _null_;
 set test.root;
 call symputx("access_token",access_token);
run;
 
%put &access_token;

So everywhere I used TOKEN-STRING in the previous code, I can now use the macro variable instead, like this:

proc http OAUTH_BEARER="&access_token" 
 url="https://server.example.com/folders/folders/@myFolder" 
 method="get" 
 out=new;
run;

Debugging with PROC HTTP

With the cURL command, you can use the -v (verbose) argument to get connection and header information. It's helpful for debugging and diagnosing trouble spots.

In SAS® 9.4M5, the DEBUG statement was added to PROC HTTP. DEBUG supports several options. I'll highlight the LEVEL= option here:

 Level= 0 | 1 | 2 | 3

Selecting 0 provides no debugging information while 3 provides the highest amount of data and messages. Base SAS 9.4 Procedures Guide includes full descriptions of each debug level.

See the HTTP Procedure documentation for additional syntax.

Generating cURL commands with Postman

If you want someone to "write your code for you," I recommend using a product like Postman to test your POST and GET commands with your API from outside of SAS. Postman is an open-source product with a super cool feature: it will produce CURL commands from successful communication with a URL.

Recommended resources

How to translate your cURL command into SAS code was published on SAS Users.

7月 302020
 

How can organizations be more resilient during an uncertain business and public health situation? Lessons learned from organizations at the front lines of crisis response reinforce the importance of good analytic governance. These leading organizations have shown us that the fundamentals of the analytic life cycle – data, discovery and [...]

Don't forget analytic fundamentals in times of crisis was published on SAS Voices by Alyssa Farrell

7月 302020
 

With COVID-19 spreading worldwide, accurate data have become more important than ever. In this blog, I share some of my favorite sources:

The Economist Reported deaths often underestimate actual deaths. One way to get at the real numbers is to compare total deaths from all causes versus the typical death rate. This “mortality tracker” plots excess deaths which is a more reliable measure than reported deaths.

Johns Hopkins University This interactive dashboard by the Coronavirus Resource Center at the Bloomberg School of Public Health shows detailed data about the pandemic worldwide.

National Public Radio These interactive graphics by NPR focus on the pandemic in the US.

SAS Institute This interactive dashboard gives a different look at global COVID-19 data.

Avi Schiffmann This webpage may be the most impressive effort by an individual person, and shows that tabular data can be profoundly thought-provoking too.

These articles are also highly recommended:

The Risks–Know Them–Avoid Them This article explains in plain language how COVID-19 spreads and how to keep yourself safe. Share this with your family.

COVID-19 Superspreader Events in 28 Countries: Critical Patterns and Lessons This fascinating article compiles data about superspreader events (SSEs) and reveals a lot about how this virus is spread.

Temporary reduction in daily global CO2 emissions during the COVID-19 forced confinement Finally, something positive: an article about the reduction in CO2 emissions due to the pandemic.

Knowledge is power. Working together we can all stay healthy.

7月 292020
 

Students in my classes often want to know about how to discover what's going on in their SAS environment. They get auditing questions from others in their organizations and don’t know how to find the answers. I honestly get a bit giddy when I can point them to SAS® software's Report Center, so it seems fitting to tout it now, as Friday is System Administrator Appreciation Day.

The Report Center, available since SAS 9.4M3, is a collection of stored processes that produce reports from data in the SAS Environment Manager Data Mart. These reports are a window into the performance and status of your SAS environment and its resources. They're samples of the types of reports you can produce using available metric data. You can also create your own reports to meet your individual requirements.

How Report Center works

The stored processes in the Report Center are created when you initialize SAS Environment Manager Extended Monitoring. However, the stored processes operate only on data that was stored in the SAS Environment Manager Data Mart by the APM or ACM ETL processes. Unless you initialize and enable one of those packages, no reports are produced.

The information in SAS Environment Manager Data Mart is the storage area for the Audit Performance Metrics (APM) and Agent Collected Metrics (ACM). APM scans the components in your SAS system for SAS server logs, SAS job logs, SAS Metadata and HTTP access logs; basically, everything SAS. The ACM collects information such as workload, CPU usage, and memory.

The current SAS Administration Fast Track course, which includes information about the Report Center, is taught on SAS 9.4M6. Before I decided to write this post, I searched and found others have provided information about the Report Center. There are several other blog posts and videos about it, so it surprises me how many I teach are unaware of the Report Center.

After generating reports in class, I have a whole new band of converts for the SAS Report Center. The great news is that it is never too late to initialize the Report Center in your SAS 9 environment.

Learn more about how to use the Report Center in the SAS Environment manager 2.5 User's Guide. Training for the SAS Report Center is found in the SAS® Platform Administration: Fast Track. Happy SysAdmin Day a couple of days early! I look forward to the opportunity to meet you in one of our upcoming classes!

TAKE THE CLASS | SAS® PLATFORM ADMINISTRATION: FAST TRACK

Report Center: SAS SysAdmin's secret weapon was published on SAS Users.