Developers

2月 252021
 

The people, the energy, the quality of the content, the demos, the networking opportunities…whew, all of these things combine to make SAS Global Forum great every year. And that is no exception this year.

Preparations are in full swing for an unforgettable conference. I hope you’ve seen the notifications that we set the date, actually multiple dates around the world so that you can enjoy the content in your region and in your time zone. No one needs to set their alarm for 1:00am to attend the conference!

Go ahead and save the date(s)…you don’t want to miss this event!

Content, content, content

We are working hard to replicate the energy and excitement of a live conference in the virtual world. But we know content is king, so we have some amazing speakers and content lined up to make the conference relevant for you. There will be more than 150 breakout sessions for business leaders and SAS users, plus the demos will allow you to see firsthand the innovative solutions from SAS, and the people who make them. I, for one, am looking forward to attending live sessions that will allow attendees the opportunity to ask presenters questions and have them respond in real time.

Our keynote speakers, while still under wraps for now, will have you on the edge of your seats (or couches…no judgement here!).

Networking and entertainment

You read that correctly. We will have live entertainment that'll have you glued to the screen. And you’ll be able to network with SAS experts and peers alike. But you don’t have to wait until the conference begins to network, the SAS Global Forum virtual community is up and running. Join the group to start engaging with other attendees, and maybe take a guess or two at who the live entertainment might be.

A big thank you

We are working hard to bring you the best conference possible, but this isn’t a one-woman show. It takes a team, so I would like to introduce and thank the conference teams for 2021. The Content Advisory Team ensures the Users Program sessions meet the needs of our diverse global audience. The Content Delivery Team ensures that conference presenters and authors have the tools and resources needed to provide high-quality presentations and papers. And, finally, the SAS Advisers help us in a multitude of ways. Thank you all for your time and effort so far!

Registration opens in April, so stay tuned for that announcement. I look forward to “seeing” you all in May.

What makes SAS Global Forum great? was published on SAS Users.

12月 172020
 

There’s nothing worse than being in the middle of a task and getting stuck. Being able to find quick tips and tricks to help you solve the task at hand, or simply entertain your curiosity, is key to maintaining your efficiency and building everyday skills. But how do you get quick information that’s ALSO engaging? By adding some personality to traditionally routine tutorials, you can learn and may even have fun at the same time. Cue the SAS Users YouTube channel.

With more than 50 videos that show personality published to-date and over 10,000 hours watched, there’s no shortage of learning going on. Our team of experts love to share their knowledge and passion (with personal flavor!) to give you solutions to those everyday tasks.

What better way to round out the year than provide a roundup of our most popular videos from 2020? Check out these crowd favorites:

Most viewed

  1. How to convert character to numeric in SAS
  2. How to import data from Excel to SAS
  3. How to export SAS data to Excel

Most hours watched

  1. How to import data from Excel to SAS
  2. How to convert character to numeric in SAS
  3. Simple Linear Regression in SAS
  4. How to export SAS data to Excel
  5. How to Create Macro Variables and Use Macro Functions
  6. The SAS Exam Experience | See a Performance-Based Question in Action
  7. How it Import CSV files into SAS
  8. SAS Certification Exam: 4 tips for success
  9. SAS Date Functions FAQs
  10. Merging Data Sets in SAS Using SQL

Latest hits

  1. Combining Data in SAS: DATA Step vs SQL
  2. How to Concatenate Values in SAS
  3. How to Market to Customers Based on Online Behavior
  4. How to Plan an Optimal Tour of London Using Network Optimization
  5. Multiple Linear Regression in SAS
  6. How to Build Customized Object Detection Models

Looking forward to 2021

We’ve got you covered! SAS will continue to publish videos throughout 2021. Subscribe now to the SAS Users YouTube channel, so you can be notified when we’re publishing new videos. Be on the lookout for some of the following topics:

  • Transforming variables in SAS
  • Tips for working with SAS Technical Support
  • How to use Git with SAS

2020 roundup: SAS Users YouTube channel how to tutorials was published on SAS Users.

12月 072020
 

In my previous blog post, I talked about using PROC CAS to accomplish various data preparation tasks. Since then, my colleague Todd Braswell and I worked through some interesting challenges implementing an Extract, Transform, Load (ETL) process that continuously updates data in CAS. (Todd is really the brains behind getting this to work – I am just along for the ride.)

In a nutshell, the process goes like this:

  1. PDF documents drop into a "receiving" directory on the server. The documents have a unique SubmissionID. Some documents are very large – thousands of pages long.
  2. A Python job runs and converts PDFs into plain text. Python calls an API that performs Optical Character Recognition (OCR) and saves off the output as a CSV file, one row per page, in the PDF document.
  3. A SAS program, running in batch, loads the CSV file with new records into a CAS table. SubmissionID is passed to the batch program as a macro variable, which is used as part of the CAS table name.
  4. Records loaded from the CSV file are appended to the Main table. If records with the current SubmissionID already exist in the Main table, they are deleted and replaced with new records.
    The Main table is queried by downstream processes, which extract subsets of data, apply model score code, and generate results for the customer.

Continuously update data process flow

Due to the volume of data and the amount of time it takes to OCR large PDFs, the ETL process runs in multiple sessions simultaneously. And here is a key requirement: the Main table is always available, in a promoted state, with all up-to-date records, in order for the model score code to pick up the needed records.

What does "promoted state" mean?

The concept of table scope, which was introduced with the first release of SAS Viya, presents a challenge. CAS tables are in-memory tables that can have one of two "scopes":

  • Session scope – the table exists within the scope of your current CAS session and drops from memory as soon as the session ends. Functionally, this is somewhat similar to the data you write to the WORK library in SAS 9 – once you disconnect, the data drops from the WORK library.
  • Global scope – the table is available to all sessions. If your session ends, you will still have access to it when you start a new session. Your colleagues also maintain access, assuming they have the necessary permissions. For the table to assume global scope, it needs to be promoted.

Common promotion techniques for a table are the DATA STEP, PROC CAS, or PROC CASUTIL. For example:

/*promote a table using DATA STEP*/
*this snippet copies a SAS 9 table into CAS and promotes in one step;
data mycaslib.my_table (promote=yes);
     set mylib.my_table;
     run;
 
/*promote using PROC CASUTIL*/
*this snippet promotes a session-scope CAS table to global scope;
proc casutil incaslib='mycaslib' outcaslib='mycaslib';
     promote casdata='my_table';
     quit;
 
/*Promote using PROC CAS*/
*same as above, this snippet promotes a session-scope table to global scope;
proc cas;
table.promote / 
     caslib='mycaslib'
     targetcaslib='mycaslib' 
     name='my_table' 
     target='my_table';
    quit;

Fun Facts About Table Promotion

You cannot promote a table that has already been promoted. If you need to promote a new version of the same table, you need to first drop the existing table and promote the new version.

To discover the current table state, use the

proc cas;
     table.tableinfo / 
     caslib='mycaslib' 
     name='main';
     quit;

If you append rows to a promoted table using the DATA STEP append option, the new rows are automatically promoted. For example, in this snippet the mycaslib.main table, which is promoted, remains promoted when the rows from mycaslib.new_rows are appended to it:

data mycaslib.main(append=yes);
     set mycaslib.new_rows;
     run;

When you manipulate a promoted table using the DATA STEP apart from appending rows, it creates a new, session-scope version of the same table. You will have two versions of the table: the global-scope table, which remains unchanged, and the session-scope version which has the changes you implemented. Even if you don't change anything in the table and simply run:

data mycaslib.my_table;
     set mycaslib.my_table;
     run;

in which mycaslib.my_table is promoted, you end up with a promoted and an unpromoted version of this table in the mycaslib library – a somewhat unexpected and hardly desired result. Appendix 1 walks through a quick exercise you can try to verify this.

As you probably guessed, this is where we ran into trouble with our ETL process: the key requirement was for the Main table to remain promoted, yet we needed to continuously update it. The task was simple if we just needed to append the rows; however, we also needed to replace the rows if they already existed. If we tried to delete the existing rows using the DATA STEP, we would have to deal with the changes applied to a session-scope copy of the global-scope table.

Initially, we designed the flow to save off the session-scope table with changes, then drop the (original) global-scope version, and finally reload the up-to-date version. This was an acceptable workaround, but errors started to pop up when a session looked for the Main table to score data, while a different concurrent session reloaded the most up-to-date data. We were uncertain how this would scale as our data grew.

PROC CAS to the rescue!

After much research, we learned the deleteRows, which allows you to delete rows directly from a global-scope table. The data is never dropped to session-scope – exactly what we needed. Here's an example:

proc cas;
     table.deleteRows /
     table={caslib="mycaslib", name="Main", where="SubmissionID = 12345"};
     quit;

In case you are wondering, the Tables action set also has an

/*1. Load new rows. SubmissionID macro variable is a parameter passed to the batch program*/
/*New rows are written to the casuser library, but it does not really matter which caslib you choose – 
   we are not persisting them across sessions*/
proc casutil;
       load file="/path_to_new_rows/New_rows_&SubmissionID..csv" outcaslib="casuser" casout="new_rows";
       quit;
/*2. Delete rows with the current SubmissionID */
proc cas;
       table.deleteRows /
       table={caslib="prod", name="Main", where="SubmissionID = &SubmissionID."};
       quit;
/*3. Append new rows*/
data mycaslib.main(append=yes);
	set mycaslib.new_rows;
	run;
/*4. Save the main table to ensure we have a disk backup of in-memory data*/
proc casutil incaslib="prod" outcaslib="prod";
	save casdata="main" replace;
	quit;

Conclusion

We learned how to continuously update data in CAS while ensuring the data remains available to all sessions accessing it asynchronously. We learned the append option in DATA STEP automatically promotes new rows but manipulating the data in a global-scope table through DATA STEP in other ways leads to data being copied to session-scope. Finally, we learned that to ensure the table remains promoted while it is updated, we can fall back on PROC CAS.

Together, these techniques enabled implementation of a robust data flow that overcomes concurrency problems due to multiple processes updating and querying the data.

Acknowledgement

We thank Brian Kinnebrew for his generous help in investigating this topic and the technical review.

Appendix 1

Try the following exercise to verify that manipulating a promoted table in DATA STEP leads to two copies of the table – session- AND global-scope.

/*copy and promote a sample SAS 9 table*/
data casuser.cars(promote=yes);
     set sashelp.cars;
     run;
/*check the number of rows and confirm that the table is promoted*/
proc cas;
     table.tableinfo / caslib='casuser' name='cars';
     quit; /*The table is promoted and has 428 rows*/
 
/*delete some rows in the promoted table*/
data casuser.cars;
     set casuser.cars;
     if make='Acura' then delete;
     run;
/*check again – how may rows does the table have? Is it promoted?*/
proc cas;
     table.tableinfo / caslib='casuser' name='cars';
     quit;  /*The table is has 421 rows but it is no longer promoted*/
 
/*reset your CAS session*/
/*kill your current CAS session */
cas _all_ terminate;
/*start a new CAS session and assign caslibs*/
cas; 
caslib _all_ assign;
 
/*check again – how may rows does the table have? Is it promoted?*/
proc cas;
     table.tableinfo / caslib='casuser' name='cars';
     quit;  /*The table is promoted and has 428 rows*/

What we see here is, manipulating a global-scope table in DATA STEP leads to duplication of data. CAS copies the table to session-scope and applies the changes there. The changes go away if you terminate the session. One way to get around this issue is instead of trying to overwrite the promoted table, create a new table, then drop the old table and promote the new table under the old table's name. Otherwise, use table.Update actions to update/delete rows in place, as described in this post.

Append and Replace Records in a CAS Table was published on SAS Users.

11月 202020
 

If you’re like me and the rest of the conference team, you’ve probably attended more virtual events this year than you ever thought possible. You can see the general evolution of virtual events by watching the early ones from April or May and compare them to the recent ones. We at SAS Global Forum are studying the virtual event world, and we’re learning what works and what needs to be tweaked. We’re using that knowledge to plan the best possible virtual SAS Global Forum 2021.

Everything is virtual these days, so what do we mean by virtual?

Planning a good virtual event takes time, and we’re working through the process now. One thing is certain -- we know the importance of providing quality content and an engaging experience for our attendees. We want to provide attendees with the opportunity as always, but virtually, to continue to learn from other SAS users, hear about new and exciting developments from SAS, and connect and network with experts, peers, partners and SAS. Yes, I said network. We realize it won’t be the same as a live event, but we are hopeful we can provide attendees with an incredible experience where you connect, learn and share with others.

Call for content is open

One of the differences between SAS Global Forum and other conferences is that SAS users are front and center, and the soul of the conference. We can’t have an event without user content. And that’s where you come in! The call for content opened November 17 and lasts through December 21, 2020. Selected presenters will be notified in January 2021. Presentations will be different in 2021; they will be 30 minutes in length, including time for Q&A when able. And since everything is virtual, video is a key component to your content submission. We ask for a 3-minute video along with your title and abstract.

The Student Symposium is back

Calling all postsecondary students -- there’s still time to build a team for the Student Symposium. If you are interested in data science and want to showcase your skills, grab a teammate or two and a faculty advisor and put your thinking caps on. Applications are due by December 21, 2020.

Learn more

I encourage you to visit the SAS Global Forum website for up-to-date information, follow #SASGF on social channels and join the SAS communities group to engage with the conference team and other attendees.

Connect, learn and share during virtual SAS Global Forum 2021 was published on SAS Users.

11月 102020
 

The code and data that drive analytics projects are important assets to the organizations that sponsor them. As such, there is a growing trend to manage these items in the source management systems of record. For most companies these days, that means Git. The specific system might be GitHub Enterprise, GitLab, or Bitbucket -- all platforms that are based on Git.

Many SAS products support direct integration with Git. This includes SAS Studio, SAS Enterprise Guide, and the SAS programming language. (That last one checks a lot of boxes for ways to use Git and SAS together.) While we have good documentation and videos to help you learn about Git and SAS, we often get questions around "best practices" -- what is the best/correct way to organize your SAS projects in Git?

In this article I'll dodge that question, but I'll still try to provide some helpful advice in the process.

Ask the Expert resource: Using SAS® With Git: Bring a DevOps Mindset to Your SAS® Code

Guidelines for managing SAS projects in Git

It’s difficult for us to prescribe exactly how to organize project repositories in source control. Your best approach will depend so much on the type of work, the company organization, and the culture of collaboration. But I can provide some guidance -- mainly things to do and things to avoid -- based on experience.

Do not create one huge repository

DO NOT build one huge repository that contains everything you currently maintain. Your work only grows over time and you'll come to regret/revisit the internal organization of a huge project. Once established, it can be tricky to change the folder structure and organization. If you later try to break a large project into smaller pieces, it can be difficult or impossible to maintain the integrity of source management benefits like file histories and differences.

Design with collaboration in mind

DO NOT organize projects based only on the teams that maintain them. And of course, don't organize projects based on individual team members.

  • Good repo names: risk-adjustment-model, engagement-campaigns
  • Bad repo names: joes-code, claims-dept

All teams reorganize over time, and you don't want to have to reorganize all of your code each time that happens. And code projects change hands, so keep the structure personnel-agnostic if you can. Major refactoring of code can introduce errors, and you don't want to risk that just because you got a new VP or someone changed departments.

Instead, DO organize projects based on function/work that the code accomplishes. Think modular...but don't make projects too granular (or you'll have a million projects). I personally maintain several SAS code projects. The one thing they have in common is that I'm the main contributor -- but I organize them into functional repos that theoretically (oh please oh please) someone else could step in to take over.

The Git view of my YouTube API project in SAS Enterprise Guide

Up with reuse, down with ownership

This might seem a bit communist, but collaboration works best when we don't regard code that we write as "our turf." DO NOT cling to notions of code "ownership." It makes sense for teams/subject-matter experts to have primary responsibility for a project, but systems like Git are designed to help with transparency and collaboration. Be open to another team member suggesting and merging (with review and approval) a change that improves things. GitHub, GitLab, and Bitbucket all support mechanisms for issue tracking and merge requests. These allow changes to be suggested, submitted, revised, and approved in an efficient, transparent way.

DO use source control to enable code reuse. Many teams have foundational "shared code" for standard operations, coded in SAS macros or shared statements. Consider placing these into their own project that other projects and teams can import. You can even use Git functions within SAS to fetch and include this code directly from your Git repository:

/* create a temp folder to hold the shared code */
options dlcreatedir;
%let repoPath = %sysfunc(getoption(WORK))/shared-code;
libname repo "&repoPath.";
libname repo clear;
 
/* Fetch latest code from Git */
data _null_;
 rc = git_clone( 
   "https://gitlab.mycompany.com/sas-projects/shared-code/",
   "&repoPath.");
run;
 
options source2;
/* run the code in this session */
%include "&repoPath./bootstrap-macros.sas";

If you rely on a repository for shared code and components, make sure that tests are in place so changes can be validated and will not break downstream systems. You can even automate tests with continuous integration tools like Jenkins.

DO document how projects relate to each other, dependencies, and prepare guidance for new team members to get started quickly. For most of us, we feel more accountable when we know that our code will be placed in central repositories visible to our peers. It may inspire cleaner code, more complete documentation, and a robust on-boarding process for new team members. Use the Markdown files (README.md and others) in a repository to keep your documentation close to the code.

My SAS code to check Pagespeed Insights, with documentation

Work with Git features (and not against them)

Once your project files are in a Git repository, you might need to change your way of working so that you aren't going against the grain of Git benefits.

DO NOT work on code changes in a shared directory with multiple team members –- you'll step on each other. The advantage of Git is that it's a distributed workflow and each developer can work with their own copy of the repository, and merge/accept changes from others at their own pace.

DO use Git branching to organize and isolate changes until you are ready to merge them with the main branch. It takes a little bit of learning and practice, but when you adopt a branching approach you'll find it much easier to manage -- it beats keeping multiple copies of your code with slightly different file and folder names to mark "works in progress."

DO consider learning and using Git tools such as Git Bash (command line), Git GUI, and a code IDE like VS Code. These don't replace the SAS-provided coding tools with their Git integration, but they can supplement your workflow and make it easier to manage content among several projects.

Learning more

When you're ready to learn more about working with Git and SAS, we have many webinars, videos, and documentation resources:

The post How to organize your SAS projects in Git appeared first on The SAS Dummy.

9月 262020
 

Welcome back to my series on securely integrating custom applications into your SAS Viya platform. My goal today is to lay out some examples of the concepts I introduced in the previous posts. As a quick recap:

  1. In the first installment of this series I shared my experiences on a customer project, where we helped achieve value-add on top of their SAS Viya platform. We built a custom application for their buyers to use at auto auctions and embedded repeatable analytics from a SAS Viya decisioning workflow. That customization now helps their buyers make data-driven decisions based on both business rules and analytical models trained by historical data from past purchases and sales. The models are re-trained by the workflow as buyers "commit" new purchases.
  2. In part two, I outlined the security frameworks (OAuth 2.0 and OpenID Connect) and main integration points for bringing custom apps and SAS Viya together. I also introduced the main decision points before setting up this integration.
  3. The most recent post detailed the OAuth flow choices for access tokens from SAS Logon used for custom application and SAS Viya integration. I outlined some suggestions on when to use each flow in order to retrieve the appropriate scoping for your app.

I would encourage you to check out the previous parts in this series before reading on so we all start on a level playing field. In this post, I will assume you have, and thus are familiar with the concepts of user- and general-scoping. Before we jump into some examples for both of those scenarios, two things to note:

  • Recall for both user-scoped and general-scoped scenarios, the Password authentication flow is technically valid, but the industry is suggesting phasing out the use of this flow (source one) (source two). Accordingly, I won't include examples of it here.
  • This post assumes that you are familiar with the SAS Administration concepts of Custom Groups, granting access to SAS Content, and creating authorization rules. If those topics are unfamiliar to you, and you will be implementing this custom application registration, please take a look at the resources linked inline.

General-scoped access tokens

This refers to scenarios where we don't want end users to have to log in (directly to SAS Viya, at least). You don't want the tokens returned by SAS Logon to your custom applications generating authorization decisions based on the individual using your application. In these cases, however, you do still want control over the endpoints/resources your app calls and which CRUD operations (which map to HTTP methods) it performs on them. The general steps are:

  • Create a Custom Group for the use of the application.
  • Grant necessary access to that Custom Group for the appropriate endpoints and resources.
  • Register a new, unique client to your SAS Viya environment, using the client_credentials grant type, and specifying the created Custom Group as the value for the authorities property.

Detailed steps

  1. As a member of your SAS Viya environment's SAS Administrators group, create a new Custom Group for your application. The naming convention of App.<your_app> is recommended. Using prefixes in this manner logically organizes Custom Groups used for similar purposes together. Consider including a description like the one pictured below.

    (Click to enlarge)

  2. Determine the necessary resource(s), content, and/or endpoint(s) application users need. Next, create the authorization rules to provide appropriate access using the Custom Group you created in the previous step. I've often found this to be an iterative process which is perfectly fine. You (a SAS Admin) can adjust the authorization rules granted to this Custom Group, and thus your app, at any time.
    _

  3. Review the general documentation provided for registering clients to your SAS Viya environment. The following is an adaptation of those instructions, please take note of the differences and be sure to include them as you see below.

    (Click to enlarge)

    For readability, the required data for the last step above was:

    {
        "client_id": "{{YOUR_APP_NAME}}",
        "client_secret": "{{YOUR_SECRET}}",
        "authorized_grant_types": "client_credentials",
        "authorities": "{{CUSTOM_GROUP_FROM_PREVIOUS_STEPS}}",
        "scope": "openid"
    }

    Where scope should include openid for authentication purposes.

    And where authorities will include your Custom Group linking this client with the authorization rules you created. This effectively grants those same permissions to any downstream API requests made by your application when it requests a token in the manner below (next step).
    _

  4. Once registered, here is an example (using Postman) of how your application requests a token using the Client Credentials flow:

    (Click to enlarge)

    _

  5. And here is an example of how your application would make an API call using that token. In this example, an authorization rule was created for this endpoint with Read (GET) permissions for the Custom Group linked to this client:

    (Click to enlarge)

Some important things to note

  • You'll want to use dynamic variables instead of hard coded values as you see in the Postman calls above. This is just for illustrative purposes.
  • If you make a GET request against the client you just registered for your application, you will not get back the client_secret value you specified upon creation. Note that as we saw in Step 4 above, your client will need this value to request it's tokens so plan accordingly.
  • In addition to any values that could change depending on where your application is deployed (including hostnames, etc.), ensure your client_id and client_secret values are secured properly. If you're using Docker, feed an environment variables file to docker-compose.yml. Make sure to include the variables file in your gitignore file. Moreover, if you're using k8s you can always use secrets. Just have a plan and never hard code these into your application.
  • When making requests to SAS Logon Manager for tokens within the context of your registered application as shown in the first screenshot above, the default timeout for the tokens returned is just shy of 12 hours. You can augment that by adding the access_token_validity property with a numeric value in seconds when you register your client. Any subsequent tokens obtained using your client will use that timeout value.
  • OAuth's Refresh Token flow isn't available in combination with the client_credentials grant type. Keep this in mind when deciding on the value to use for the access_token_validity property. You may want to consider adding logic to your application to request a new token before the current one expires. Parsing out the value of the expires_in property dynamically from the original token request response will be helpful in accomplishing that. Keep in mind you will want enough of a time buffer if your application makes a string of endpoint requests or any long-running requests, so your process does not get interrupted by an expired token.

_


User-scoped access tokens

This refers to scenarios where it does make sense for your application's end users to log in to SAS. All subsequent API calls into SAS Viya resources your application makes should be within the context of what they, as an individual user, is allowed to do/see/access in SAS Viya. The authorization decisions for those requests will be based on the union of anything the logged-in user could do while logged into a native SAS Viya UI (see autoapprove below) and any authorization rules that were explicitly granted to the Custom Group(s) selected while registering this client. A SAS Administrator can place all the users of your application in that Custom Group, ensuring a similar level of functionality for those users, while still obeying any more-granular rules that might apply, such as on specific data or content. The general steps are:

  • Create a Custom Group for the use of the application
  • Grant necessary access to that Custom Group for the appropriate endpoints and resources.
  • Register a new, unique client to your SAS Viya environment, using the authorization_code grant type, specifying the created Custom Group as the value in the scope property, and some additional specific properties shown below
  • Include logic in your application to handle the redirection to SAS Logon Manager when needed and to temporarily store authorization codes and OAuth tokens per session/user.

Detailed steps: prepping and registering the client

  1. Perform steps 1 and 2 from the General-scoped tokens section above.
    _

  2. Review the general documentation provided for registering clients to your SAS Viya environment. The following is an adaptation of those instructions, please take note of the differences and be sure to include them as you see below.

    (Click to enlarge)

    For readability, the required data for the last step above was:

     {
        "client_id": "{{YOUR_APP_NAME}}",
        "client_secret": "{{YOUR_SECRET}}",
        "authorized_grant_types": ["authorization_code", "refresh_token"],
        "scope": ["openid", "{{CUSTOM_GROUP_FROM_PREVIOUS_STEPS"],
        "redirect_uri": ["{{URL_SASLOGON_SHOULD_REDIRECT_BACK_TO}}/callback"],
        "autoapprove": "{{TRUE_OR_FALSE_IN_LOWCASE}}" 
    }

    Where scope should include openid for authentication purposes and, for this authorization flow, must also include the Custom Group you created. This is what links your client to any authorization rules you created, effectively granting those same permissions to any downstream API requests made by your application.

    And where redirect_uri includes the URL SAS Logon Manager sends your users back to after they authenticate. It's formatted in a list above in case you want to include both HTTP and HTTPS protocols for development purposes. This is helpful in avoiding edits to this client down the road. Take note of the inclusion of the /callback postfix in the screenshot above. We'll see more on this later.

    And where autoapprove is either true or false, depending on whether or not you want to present the user with a list of Custom Groups they need to approve or deny before proceeding. My personal opinion is to stick with true, not only because it more-closely mimics the behavior already present with native SAS Viya applications. In addition, the user likely won't know the implications of selecting (or not selecting) certain Custom Groups. This could lead to downstream authorization issues for your application.
    _

Detailed steps: partial example for your app's logic

Your app has several requirements pertaining to authentication and authorization. First, it needs to know when to redirect users to SAS Logon Manager (when they're not already logged in or if their session expired). The app also must handle the authorization code returned to your user. Further, it needs to request an OAuth token for your user and how to use that token to make downstream requests to other APIs. Finally, your app needs to revoke tokens when necessary.

This is not a complete example and exactly how this is implemented depends heavily on the language your application was developed in. I don't want to distract from the topic at hand, so I'm not going to include complete code blocks. The application used for this example is a Javascript React web UI and I'll focus on the main order of operations involved in getting this to work. Therefore, I've intentionally removed some syntax to illustrate this in a more agnostic way.

  1. User navigates to our application's home page. Logic determines that a boolean state variable we're setting called isAuthenticated is false (default), so it redirects to the SAS Logon page using the URL below. Note that all items in {{ }} are substituted with environment variables rather than hardcoded. Recall, in the previous step when we registered the client, I mentioned you might want to include /callback after the URL to your application. When you're developing the routes in your code logic, it helps to differentiate between someone who has navigated directly to your / or /home page versus a user that's been redirected there from SAS Logon Manager. You'll see why in the next step.
    https://{{VIYA_HOSTNAME}}/SASLogon/oauth/authorize/response_type=code&client_id={{CLIENT_ID}}&redirect_uri={{URL_POSTFIXED_WITH_/callback}}
  2. This automagically presents the user with the same SAS Logon screen they see when accessing a native SAS Viya application. After they successfully authenticate, the URL redirects them back to our application (we included the /callback postfix in the previous step, so we see that in the redirected URL below) with an extra URL parameter like this (the code will of course be dynamic):
    https://{{YOUR_APPS_HOSTNAME}}/callback?code=pe3vyay7LJ
  3. So now, we substring off of the value of the user's current URL path and store that code in a variable, say code, temporarily. Next, the conditional logic will pick up the fact the user does have a code but isAuthenticated is false, so we kick off another function, feeding the code as an argument.
    _
  4. This function makes the second call to SAS Logon Manager; this time to exchange the user's code for an OAuth token by making a request to:
    https://{{VIYA_HOSTNAME}}/SASLogon/oauth/token/

    _
    With headers similar to the snippet below, again where anything in {{ }} is dynamically set:

    -H "Accept: application/json"
    -H "Content-Type: application/x-www-form-urlencoded"
    -u "{{CLIENT_ID}}:{{CLIENT_SECRET}}" -d "grant_type=authorization_code&code={{CODE}}"
  5. If the request is successful, we then call another function, which uses the js-cookie package in our case, to set a new piece of information in the user's cookie called access_token. This value gets pulled from the result of the previous request. At the same time, we update our isAuthenticated state variable to true.
    _
  6. Now that the state has changed (isAuthenticated is true), the page gets re-rendered in typical React fashion. If our URL included /callback then we know the user came from SAS Logon, and our application logic redirects the user to the main /home page of our application. Now we have the user's OAuth token stored in a cookie for them.

At this point, I think we've seen enough examples to know how to make downstream requests with the token by adding an Authorization header and setting it's value to Bearer {{TOKEN}}, so I'll leave those pieces out from here. Don't forget each of those requests will be made for your application's individual logged-in users, so their unique identities will need to be able to access the endpoints/resources in SAS Viya through appropriate authorization rules.

On a similar note, and because this is getting really long 🙂 , it feels a little out-of-scope to include details on the refresh token logic. Just know, it follows a very similar pattern as above. If you've included refresh_token in the authorized_grant_types list when registering your client, then subsequent token requests made within the context of your client automatically include a refresh_token property in the response you can parse out and use to get a new bearer token. The logout logic includes removing the user's cookie.

Just one more thing: if you have a multi-machine SAS Viya environment then using {{VIYA_HOSTNAME}} might have been confusing. Check your Ansible inventory.ini file and look for the machine specification for the hostgroup [CoreServices].

Thank you!

Whew. That was a doozy. Thanks for sticking with me through this post and the series! I hope this content gives you a better understanding of how to integrate your custom applications with SAS Viya and how to do it securely. Remember to evaluate each application's needs and intended use separately to choose the most appropriate type of scoping. Then use the guidance presented here to choose an OAuth flow so your application makes HTTP REST API calls into the SAS Viya endpoints and resources it needs. Feel free to reach out if you have any questions.

9月 162020
 

There are three types of visualization APIs defined in the SAS Viya REST API reference documetation: Reports, Report Images and Report Transforms. You may have seen the posts on how to use Reports and Report Images. In this post, I'm going to show you how to use the Report Transforms API. The scenario I am using changes the data source of a SAS Visual Analytics report and saves the transformed report.

Overview of the Report Transforms API

The Report Transforms API provides simple alterations to SAS Visual Analytics reports, and it uses the 'application/vnd.sas.report.transform' media type for the transformation (passed in the REST API call header). When part of a request, the transform performs editing or modifications to a report. If part of a response, the transform describes the operation performed on the report. Some typical transformations include:

  • Replace a data source of a report.
  • Change the theme of a report.
  • Translate labels in a report.
  • Generate an automatic visualization report of a specified data source and columns.

To use the Transforms API, we need to properly set the request body and some attributes for the transform. After the transform, the response contains the transformed report or a reference to the report.

Prepare the source report

This step is very straight-forward. In SAS Visual Analytics, create a report and save it to a folder (instructions on creating a report are found in this video). For this example, I'll use the 'HPS.CARS' table as the data source and create a bar chart. I save the report with name 'Report 1' in 'My Folder'. I'll use this report as the original report in the transform.

Generate the request body

I will use PROC HTTP to call the Transforms API using the 'POST' method and appending the URL with '/reportTransforms/dataMappedReports'. The call needs to set the request body.

  1. Get the ReportURI: In an earlier post I outlined how to get the reportURI via REST API, so I won't go into details. If you'd like an easy way, try this: in SAS Visual Analytics, choose 'Copy Link…'item from the menu. In the pop-up dialog, expand the 'Options' and choose 'Embedded Web Component', and you see there is a string in the form reportUri='/reports/reports/…', that's it. In the request body, we set the string to the 'inputReportUri' to specify the original report - the 'Report 1'.
  2. Report URI from SAS Visual Analytics

  3. Decide on changes to the data source: Here I’d like to change the data source from ‘HPS.CARS’ to ‘CASUSER.CARS_NEW’. The new table uses three columns from ‘HPS.CARS’ as mapped below.
  4. Select columns to include in new table

  5. Specify the data sources in the request body: The request requires two data sources, 'original' and 'replacement', respectively, representing the data sources in original report and the transformed report. Note that the 'namePattern' value is used to enumerate the way of identifying the data source. If it is set to 'uniqueName', the data source is identified by its unique data item name in the XML file of the report. If it is set to 'serverLibraryTable', the data source is identified by the CAS Server, CAS Library and Table names together. The snippets below show the data source section in the request body. I like to use the 'serverLibraryTable' to specify the data source for both original and transformed report, which is clear and easy.
  6. /* data source identification for original report */
      {
        "namePattern": "serverLibraryTable",
        "purpose": "original",
        "server": "cas-shared-default",
        "library": "HPS",
        "table": "CARS"
      }
     
    /* data source identification for transformed report */
      {
        "namePattern": "serverLibraryTable",
        "purpose": "replacement",
        "server": "cas-shared-default",
        "library": "CASUSER",
        "table": "CARS_NEW",
        "replacementLabel": "NEW CARS",
        "dataItemReplacements": [
          {
            "originalColumn": "dte",
            "replacementColumn": "date"
          },
          {
            "originalColumn": "wght",
            "replacementColumn": "weight"
          },	
          {
            "originalColumn": "dest",
            "replacementColumn": "region"
          }
        ]
      }

    Set more attributes for transform

    Besides the request body, we need to set some other attributes for the transform API when changing the data source. These include 'useSavedReport', 'saveResult', 'failOnDataSourceError' and 'validate'.

    • useSavedReport specifies whether to find the input (original) report as a permanent resource. Since I am using the saved report in the repository, I will set it to true.
    • saveResult specifies to save the transformed report permanently in the repository or not. I am going to save the transformed report in the repository, so I set it to true.
    • failOnDataSourceError specifies whether the transform continues if there is a data source failure. The default value is false, and I leave it as such.
    • The validate value decides if the transform will perform the XML schema validation or not. The default value is false, and I leave it is as such.

    Decide on a target report and folder

    I'll save the transformed report with the name 'Transformed Report 1' in the same folder as the original 'Report 1'. I set the 'resultReportName' to 'Transformed Report 1', and set the 'resultReport' with 'name" and 'description' attributes. I also need to get the folderURI of the 'My Folder' directory. You may refer my previous post to see how to get the folderURI using REST APIs.

    Below is the section of the settings for the target report and folder:

    "resultReportName": "Transformed Report 1",
    "resultParentFolderUri": "/folders/folders/cf981702-fb8f-4c6f-bef3-742dd898a69c",
    "resultReport": {
    			    "name": "Transformed Report 1",
    			    "description": "TEST report transform"
    			}

    Perform the transform

    Now, we have set all the necessary parameters for the transform and are ready to run the transform. I put my entire set of code on GitHub. Running the code creates the 'Transformed Report 1' report in 'My Folder', with the data source changing to CASUSER.CARS_NEW', containing the three mapped columns.

    Check the result

    If the API failed to create the transformed report, the PROC SQL statements displays an error code and error message. For example, if the replacement data source is not valid, it returns errors similar to the following.

    Invalid data source error message

    If the API successfully creates the transformed report, “201 Created” is displayed in the log. You may find more info about the transformed report from the response body of tranFile from PROC HTTP. You can also log into the SAS Visual Analytics user interface to check the transformed report is opened successfully, and the data sources are changed as expected. Below is the screenshot of the original report and transformed report. You may have already noticed they use different data sources from data labels.

    Original and transformed reports in SAS Visual Analytics

    Finally

    There are a wealth of other transformations available through the Report Transform APIs. Check out the SAS Developer site for more information.

Using REST API to transform a Visual Analytics Report was published on SAS Users.

9月 152020
 

This is the third installment of a series focused on discussing the secure integration of custom applications into your SAS Viya platform. I began the series with an example of how a recent customer achieved value-add on top of their existing investments by building a custom app for their employees and embedding repeatable analytics from SAS Viya into it. Then in the second part, I outlined the security frameworks and main integration points used to accomplish that.

Key takeaways from the previous post

As a quick recap, in the last post I covered how to integrate custom-written applications with the SAS Viya platform using scoped access tokens from SAS Logon Manager. I outlined the returned access tokens need to be included with the REST calls those custom apps make to SAS Viya endpoints and resources. The access tokens contain the necessary information SAS Viya will use to make authorization decisions for every call your app makes. Administrators of your SAS Viya environment have complete control over the rules that determine the outcome of those decisions. I also introduced the primary decision you need to make before you begin integrating your custom app into your SAS Viya environment: whether those access tokens will be scoped in the context of your application (general-scoped) or in the context of each logged-in user of your application (user-scoped).

Now let's go a bit deeper into the authorization flows for both scenarios. TL;DR at the bottom.

Quick disclaimer: The rest of this post assumes you are familiar with the SAS Administration concepts of Custom Groups, granting access to SAS Content, and creating authorization rules. If those topics are unfamiliar to you and you will be implementing this custom application registration, please take a look at the resources linked inline.

General-scoped access tokens

First, when I say "general-scoped access" here, I mean cases where you don't want end users to have to log in (directly to SAS Viya, at least). You don't want the tokens returned by SAS Logon to your custom applications generating authorization decisions based on the individual using your application. In these cases, however, you do still want control over the endpoints/resources your app calls and which CRUD operations it performs on them. Any situation where you'd like to expose some content you've created in SAS Viya to a large scale of users, particularly those that wouldn't have valid login credentials to your SAS Viya environment, fall in this category. There are two main “flows” available to make this happen. Here, “flow” is a term used to describe the process for accessing a protected resource within the OAuth framework.

Password flow

I'll won't go into much detail on this one because the folks behind OAuth 2.0 are suggesting the phase-out of the use of this flow (source one) (source two). Though it technically works for this use case, it does so by essentially using a "service account" so all REST calls your application makes are within the scope of a single UID. You could create specific authorization grants for that user (or a group it's a member of) to make this functional, but it requires that you include the username and password as clear text URL parameters when you request your token. It's just not a great idea and not secure to send this over the wire even if you are using HTTPS. It also creates another sensitive environment variable to manage and, worse, I have even seen this hard coded it into custom application logic.

Client Credentials flow

Interestingly enough, the combination of the security concerns with the Password flow, while still needing to provide specific access levels for a custom application our team was building, is how I found myself wanting to write this series. If you're reading this, you've likely checked out the page on how to register clients to SAS Logon Manger on developer.sas.com. These instructions do not cover the Client Credintials flow though Mike Roda's SAS Global Forum paper, OpenID Connect Opens the Door to SAS Viya APIs, does provide guidance on the client credentials grant type. Remember, these types of scenarios don't warrant concern for which individual users are making the calls into SAS Viya; however, you still will want to restrict which calls your app makes, so follow least-privilege access best practices.

The secret sauce to making this work is registering your application to SAS Logon Manager using a specific, optional key called authorities and setting its value to be a list of custom group(s) giving your application the access to the proper resources and endpoints. Here's where some prior knowledge of creating authorization rules for specific URIs using endpoint patterns and specific principals comes in handy. Check out the documentation linked above if those are unfamiliar terms. We'll see examples of how to implement this flow in the next post in this series.

The diagram below illustrates this flow. Viewing hint: click on the photo to focus on it, and click on the full-screen button to enlarge it.

Figure 1: Visual representation of the Client Credentials OAuth flow

User-scoped access tokens

As a quick recap, here I'm talking about the cases where it does make sense for your application's end users to log in to SAS. All subsequent API calls into SAS Viya resources your application makes should be within the context of what they as an individual user is allowed to do/see/access in SAS Viya. For example, if a user is not in the SAS Administrator's group, then you'll want to ensure any REST calls your custom app makes for that user that would require him to be in the SAS Administrator's group will indeed fail.

Admittedly, this takes more work on the application development side to implement; however, as far as creating and registering the client for your application, this isn't any more complicated than what I've already outlined in this post. By the nature of the flow I'll outline in a minute, the application developer is going to need to add additional logic to handle sessions and/or cookies to obtain, store, check, update, and eventually revoke individual user's authorization codes. When using SAS Viya applications this all occurs behind the scenes. For example, when you first navigate to SAS Visual Analytics (VA) you are re-directed to SAS Logon Manager to log in and get your user-scoped access token. Then you are redirected back to VA with your token. In turn, VA and the supporting services use this token for any subsequent REST calls to other SAS Viya endpoints and resources. When creating custom-developed apps you integrate with SAS Viya for user-scoped access, we need to recreate that baked-in behavior. This is the process I outline below.

Authorization Code flow

A shortened-for-readability version of how this works is:

    1. The user navigates to your application. If they don't already have a valid session with SAS Logon Manager redirection sends them to the same screen they would see when accessing native SAS Viya applications.
      _
    2.  They enter credentials and log in. Then depending on how the client was registered, either:
      • the user manually approves their group memberships which SAS Logon Manager presents to them (thus all the authorization grants for those group(s)). An example of this is seen when logging into SAS Viya and being presented with the option to accept or deny your SAS Administrators group privileges, or
      • their group memberships will be automatically approved for them. This option is more SAS Viya-esque in that, when you log in as a regular user you're not asked to accept your role in any Custom Groups. It is implicitly allowed behind the scenes, so you needn't know what each Custom Group allows. Rather, it is assumed that a SAS Administrator has provided you the right memberships.
      • Note that we will see an example of how to toggle these two options in the next post in this series._
    3. If the user credentials are valid, SAS Logon Manager responds with a user-scoped authorization code. We'll see how to use redirects to get it back to your application in the next part of this series. Your application needs to retrieve and temporarily store the authorization code which will be in a URL parameter on the redirect.
      _
    4. Your application will then need to make another request to SAS Logon Manager on behalf of your user, this time trading the code for an OAuth access token. This is where your apps ability to handle sessions and/or cookies, regardless of programming and code language, starts to come in handy.
      _
    5. Finally, when your user interacts with some part of your application that needs to make an API call to a SAS Viya endpoint or resource, your application needs to pass their user-scoped OAuth token in with the request. Remember, it's a good idea to include some logic to check for expired tokens and be able to refresh and revoke them when necessary as well.

Joe Furbee covers the authorization code process in detail in Authentication to SAS Viya: a couple of approaches. The whole point of using this flow in this manner is to ensure that SAS Logon Manager is responsible for handling individual user credentials rather than your application, which by contrast would then require submitting them over the wire. The diagram below represents the flow.

Figure 2: Visual representation of the Authorization Code OAuth flow

Password flow

In fairness, this technically works here as well, but we already know the drill.

_

Summing it all up (or TL;DR)...

My goal between this post (and in part 2) was to introduce the necessary planning steps prior to registering a custom-developed application to SAS Logon Manager, so it can make REST API calls to SAS Viya endpoints and resources. To recap, the steps are:

1.  Decide whether users should have to log into SAS Viya or not.

    • Is your application serving up generic or "public" content that anyone using your app should have the same level of access to? Note that in this case you could still have a layer of custom login logic to restrict user access to your application, but I'm referencing scenarios where it doesn't make sense for users to log into SAS Viya, especially if they aren't in the configured user base with valid credentials. If you find yourself in this scenario, then proceed to Step 2A.
      _
    • Or, do you want any application API calls to SAS Viya to be within the context of what that individual logged-in user should be able to do? Then proceed to Step 2B.

2A.  Consider using the Client Credentials flow. Set up the security principal(s) and authorization rule(s) to give your application the appropriate permissions to the needed resources and then register your client with the relevant settings.

2B.  In this case, you should consider using the Authorization Code flow. Enhance your custom application's ability to request and handle user- and session-specific authorization codes and tokens, substituting them into subsequent API calls when needed, and then register your client with the relevant settings.

_
In either case, I'll cover examples in the next post in this series.

Building custom apps on top of SAS Viya, Part Three: Choosing an OAuth flow was published on SAS Users.

9月 092020
 

Welcome back! Today, I continue with part 2 of my series on building custom applications on SAS Viya. The goal of this series is to discuss securely integrating custom-written applications into your SAS Viya platform.

In the first installment of this series, I outlined my experiences on a recent project. In that example, our team worked with a pre-owned auto mall to create a custom application for their buyers to use at auction, facilitating data-driven decisions and maximizing potential profits. Buyers use the app's UI to select/enter the characteristics of the specific car on which they'd like to bid. Those details are then sent via an HTTP REST call to a specific API endpoint in the customer's SAS Viya environment which exposes a packaged version of a detailed decision flow. Business rules and analytic models evaluate the car details as part of that flow. The endpoint returns a maximum bid recommendation and other helpful information via the custom applications's UI.

If you haven't already, I encourage you to read part 1 for more details.

If the concept of exposing endpoints to custom applications elicits some alarming thoughts for you, fear not. The goal of this post is to introduce the security-related considerations when integrating custom applications into your SAS Viya environment. In fact, that is the topic of the next two sections. Today we'll take a look at the security underpinnings of SAS Viya so we can start to flesh out how that integration works. In part 3, we'll dive into more specific decision points. To round things out, the series finishes with some examples of the concepts discussed in parts 1-3.

Looking a little deeper

So how does SAS Viya know it can trust your custom app? How does it know your app should have the authority to call specific API endpoints in your SAS Viya environment and what HTTP methods it should allow? Moreover, who should be able to make those REST calls, the app itself or only individual logged-in users of that app? To answer these questions, we need to understand the foundation of the security frameworks in SAS Viya.

Security in SAS Viya is built around OAuth 2.0 (authorization) and OpenID Connect (authentication), both of which are industry-standard frameworks. The details of how they work, separately and together, are out of the scope of this post. We will, however, delve into a few key things to be aware of within their implementation context for SAS Viya.

First, SAS Logon Manager is the centerpiece. When a client (application) requests access to endpoints and resources, they must first communicate with SAS Logon Manager which returns a scoped access token if it was a valid request. This is true both within SAS Viya (like when internal services hosting APIs communicate with each other) and for external applications like we’re talking about here. The client/application subsequently hands that token in with their request to the desired endpoint.

Workflow: applications requesting access to SAS Viya endpoints

The long and short of it is that we don’t want users of our custom applications to see or manipulate anything they shouldn’t, so these tokens need to be scoped. The authorization systems in SAS Viya need to be able to understand what the bearer of that specific token should be able to do. For a detailed overview of how SAS Viya services leverage OAuth and OpenID Connect tokens for authentication, I'd recommend reading this SAS Global Forum paper from SAS' Mike Roda.

Later in this series, we'll see examples of obtaining a scoped token and then using it to make REST calls to SAS Viya API endpoints, like the auto mall's custom app does to return recommendations for its users. For now, suffice it to say the scope of the token is the critical component to answering our questions regarding how SAS Viya returns specific responses to our custom application.

The glue that binds the two

We've established that native SAS Viya applications and services register themselves as clients to SAS Logon Manager. When a client requests access to an endpoint and/or resource, they must first communicate with SAS Logon Manager which returns a scoped access token if it was a valid request. If the request was unauthorized, it receives a 403 unauthorized (sometimes 401) response. Your SAS Viya environment's administrator has complete control over the rules that determine the authorization decision. We can register additional custom applications, like the auto mall's auction bidding app, to SAS Logon Manager so they too can request scoped tokens. This provides our custom applications with fine-grained access to endpoints and content within SAS Viya. But we’ve already hit a fork in the road. We need to understand the purpose of the application we’re looking to build, so we can decide if the access should be user-scoped or general-scoped.

In the pre-owned auto mall's custom application, a decision and modeling flow is exposed by way of a SAS Micro Analytics Score (MAS) module at a specific HTTP REST API endpoint. All the users of the application have the same level of access to the same endpoint by design; there was no need for differentiated access levels. This is considered an instance of general-scoped access. Abiding by a least-privilege approach, the app should only access certain data and endpoints in SAS Viya, that is, only what it needs to rather than having free reign. In this case, the appropriate level of access can be granted to the registered client (custom application) itself. The result is an access token from SAS Logon Manager scoped within the context of the application.

In other applications you might find it necessary to make more sensitive, or user-specific resources available. In this instance, you'll want to make all authorization decisions within the context of the individual logged-in user. For these situations, it is more prudent to redirect the user directly to SAS Logon Manager to log in and then redirect them back to your application. This results in SAS Logon Manager returning an access token scoped within the context of the logged-in user.

What's next?

Once you've decided which of these two scenarios most closely fits your custom application's needs, the next step is to decide on a flow (or grant) to provide the associated type of access token, general- or user-scoped. Here “flow” is a term used to describe the process for accessing a protected resource within the OAuth framework and that's what I'll cover in the next post in this series.

Building custom apps on top of SAS Viya, Part Two: Understanding the integration points was published on SAS Users.

8月 312020
 

In this multi-part series we're going to explore a real-life example of operationalizing your analytics and then quickly dive into the technical details to make it happen. The phrase Operationalize your Analytics itself encompasses a framework for realizing timely and relevant business value from models, business rules, and algorithms. It also refers to the development-to-deployment life cycle commonly used in the ModelOps space. This series is geared towards walking you through a piece of the puzzle, exposing your analytics as REST APIs, but to get a better feel for the whole picture I would encourage taking a look at these resources:

Fair warning: I get detailed here. This is a journey. I'm not trying to pitch the story for the next Netflix original, but I want to outline business challenges companies face and how they can utilize technology to solve these problems. The more background you have, the more the solution resonates. I also want to have a little fun and give props to my champion salesman.

Framing the use case

Our goal in this first part of the series is to better understand the value of building custom applications on top of SAS Viya. To illustrate my point, I'm going to use Tara's Auto Mall, a used car buying and selling business I hypothetically own. We incorporated the business back in 1980 when neon and hairspray were king and MTV played music videos. Kurt, pictured at our lot, was my best salesman.

This man knew how to spot a great buy. Every auction we'd send him to looking for cars, he'd find the gems that just needed a little polishing, a little clean up, and knew exactly how much to pay for them. Sure, we still bust on him today for a couple of heaps he bought that we never made a dime off, but for the most part he knew how not to get caught up in the heat of the auction. It's an easy thing to do while just trying to outbid the others that you end up paying too much for a car. After all, we've got to turn a profit to keep the lights on and we do that by bringing used cars from auction to our lot for resale. Over the years, Kurt learned which makes and models we could get the best profit margins out of, while also learning how to spot cars that might end up racking up too many unexpected costs. "Salt and sun damage will take a huge bite out of your commission" he'd always tell the new guys.

But these days Kurt, and most of the other great salesman I've had over the years, spends his days fishing in the middle of nowhere, happily retired. Sure, I've got a few folks still on the team that really know a good buy when they see it, but more and more often we're losing money on deals. As hard as these new kids work, they just don't have the years of experience to know what they're getting into at auction.

Sometimes a car will come up that looks like it's in great shape. Clean and well-kept, low miles even, and everyone starts spouting off bids, one right after the other. That'll make just about anyone think "they must know something I don't know, I've got to win this one!" But unless you really know what to look for, like Kurt did, that good-looking deal might end up being a dud. And while Kurt had experience under his belt to help his eye for that, keeping track of the good buys and the bad ones, learning from all those lessons - that's the best yard stick for comparisons when you're trying to make a quick call at auction. How much have we been able to sell cars with similar characteristics for before? Have they ended up costing us more money to fix up than they're worth?

Needing to modernizing our business

Okay, you get it. It's a challenge many are actively facing: a retiring workforce taking their years of experience with them, combined with agile competitors and market disrupters. Not only to better compete but also to cut down on poor "gut" decisions, Tara's Auto Mall decides to build an app. This way all their employees can use the app at auction to help make the best possible decisions for which cars to bid on and how much is too much before overpaying.  Over the years they've bought a lot of cars, thousands, and no one can really keep track of all that data in their head to know how to truly evaluate a good buy.

The buying team should be able to pull up this custom app on their phone, put in the details for the car they're thinking about making a bid on, and almost instantly get back a suggestion on whether or not they should bid at all and if so, a maximum bid amount and some other helpful information. Those results should be based on both the historical data from previous sales and from business logic. And they should be able to record how much each car up at auction went for and it's characteristics, whether they won it or not, so we can feed that back into our database to make the models even more accurate over time.

How can Tara's Auto Mall improve?

Now that we've got the main requirements for the application that Tara's Auto Mall needs to build so the buying team can make data-driven decisions at auction, we can start to hash out how those recommendations will be generated. The solution they decided on is to use SAS Viya to create a decision flow to evaluate used cars up for bidding at auction by using that specific car's details as inputs and first checking them against some business rules that they've come to trust over time. This adds a layer of intelligence into the decision-making process that models alone cannot provide.

We heard a bit about the lessons learned over time, like looking for salt and sun damage, so we can layer this into the recommendation logic by checking for certain states where that type of damage is typical. This allows us to add more "human knowledge" into the computer-driven process and automatically alert the buyer that they should be aware of this potential damage. That decision flow then uses both supervised and unsupervised models to evaluate the car's unique characteristics against previous sales, the profits from them, and market data, to generate a maximum bid suggestion.

Here's the fun part

During an auction we don't have time to wait for some batch process to give recommendations and we don't want our buyers to have to log in and enter values into a written program either. Instead, we can publish the entire decision flow as an API endpoint to the SAS Micro Analytic Service (MAS), a “compile-once, execute-many-times” service in SAS Viya. Then we can have our custom application make REST calls to that endpoint with the specific car details as inputs, where it will execute against a packaged version of that entire flow we outlined. The result is a recommendation from both the business rules and models.

The SAS decisioning process

The app is part of the SAS decisioning process, and evolves over time. As more sales are added to the database through the app, the solution automatically retrains the recommendation model. It yields more-accurate suggestions as the data changes and then republishes to MAS so our custom app can automatically take advantage of the updated model.

I want to pause here and note that, while this example is for an application making REST calls to MAS, that is really the tip of the iceberg. One of the greatest advantages to the modernization of the SAS platform with SAS Viya is the sheer number of underlying services that are now exposed as API endpoints for you to incorporate into your larger IT landscape and business processes. Check out developer.sas.com to take a look.

What's next?

Tara's Auto Mall has now decided on the custom application to help their buyers at auction and the solution to building repeatable analytics and using that to provide near-real-time recommendations. Next, we need to work through integrating the custom app into their SAS Viya environment so they can embed the recommendations.

Questions from the author

While you might not have gotten to work with Kurt, Tara's Auto Mall's best and most fashionably-dressed salesman, is your business experiencing similar challenges? Do you have homegrown applications that could benefit from injecting repeatable analytics and intelligent decision making into them?

 

Building custom apps on top of SAS Viya, Part One: Where's the value? was published on SAS Users.