Visual Analytics

7月 082020
 

The current state of policy enforcement during an infectious disease pandemic is mostly reactive. Public health officials track changes in active cases, identify hot-spots and enforce containment policies primarily based on geographic proximity. By combining telecommunications data -- which we turn into mobility information -- with public health data of [...]

Mobility tracing: Helping local authorities in the fight against COVID-19 was published on SAS Voices by Carlos Pinheiro

4月 282020
 

With increasing interest in Continuous Integration/Continuous Delivery (CI/CD), many SAS Users want to know what can be done for Visual Analytics reports. In this article, I will explain how to use Python and SAS Viya REST APIs to extract a report from a SAS Viya environment and import it into another environment. For those trying to understand the secret behind CI/CD and DevOps, here it is:

What you do tomorrow will be better than what you did yesterday because you gained more experience today!

About Continuous Integration/Continuous Delivery

If you apply this principle to code development, it means that you may update your code every day, test it, deploy it, and start the process all over again the following day. You might feel like Sisyphus rolling his boulder around for eternity. This is where CI/CD can help. In the code deployment process, you have many recurrent tasks that can be automated to reduce repetitiveness and boredom. CI/CD is a paradigm where improvements to code are pushed, tested, validated, and deployed to production in a continuous automated manner.

About ModelOps and AnalyticOps

I hope you now have a better understanding of what CI/CD is. You might now wonder how CI/CD relates to Visual Analytics reports, models, etc. With the success of DevOps which describes the Development Operations for software development, companies have moved to the CI/CD paradigms for operations not related to software development. This is why you hear about ModelOps, AnalyticOps... Wait a second, what is the difference between writing or generating code for a model or report versus writing code for software? You create a model, you test it, you validate it, and finally deploy it. You create a report, you test it, you validate it, and then you deploy it. Essentially, the processes are the same. This is why we apply CI/CD techniques to models, reports, and many other business-related tasks.

About tools

As with many methodologies like CI/CD, tools are developed to help users through the process. There are many tools available and some of them are more widely used. SAS offers SAS Workflow Manager for building workflows to help with ModelOps. Additionally, you have surely heard about Git and maybe even Jenkins.

  • Git is a version control system that is used by many companies with some popular implementations: GitHub, GitLab, BitBucket.
  • Jenkins is an automation program that is designed to build action flows to ease the CI/CD process.

With these tools, you have the needed architecture to start your CI/CD journey.

The steps

With a basic understanding of the CI/CD world; you might ask yourself: How does this apply to reports?

When designing a report in an environment managed by DevOps principles, here are the steps to deploy the report from a development environment to production:

  1. Design the report in development environment.
  2. Validate the report with Business stakeholders.
  3. Export the report from the development environment.
  4. Save a version of the report.
  5. Import the report into the test environment.
  6. Test the report into the test environment.
  7. Import the report into the production environment.
  8. Monitor the report usage and performance.

Note: In some companies, the development and test environments are the same. In this case, steps 4 to 6 are not required.

Walking through the steps, we identify steps 1 and 2 are manual. The other steps can be automated as part of the CI/CD process. I will not explain how to build a pipeline in Jenkins or other tools in this post. I will nevertheless provide you with the Python code to extract, import, and test a report.

The code

To perform the steps described in the previous section, you can use different techniques and programming languages. I’ve chosen to use Python and REST APIs. You might wonder why I've chosen these and not the sas-admin CLI or another language. The reason is quite simple: my Chinese zodiac sign is a snake!

Jokes aside, I opted for Python because:

  • It is easy to read and understand.
  • There is no need to compile.
  • It can run on different operating systems without adaptation.
  • The developer.sas.com site provides examples.

I’ve created four Python files:

  1. getReport.py: to extract the report from an environment.
  2. postReport.py: to create the report in an environment.
  3. testReport.py: to test the report in an environment.
  4. functions.py: contains the functions that are used in the other Python files.

All the files are available on GitHub.

The usage

Before you use the above code, you should configure your SAS Viya environment to enable access to REST APIs. Please follow the first step from this article to register your client.

You should also make sure the environment executing the Python code has Python 3 with the requests package installed. If you're missing the requests package, you will get an error message when executing the Python files.

Get the report

Now that your environment is set up, you can execute the getReport code to extract the report content from your source environment. Below is the command line arguments to pass to execute the code:

python3 getReport.py -a myAdmin -p myAdminPW -sn http://va85.gel.sas.com -an app -as appsecret -rl "/Users/sbxxab/My Folder" -rn CarsReport -o /tmp/CICD/

The parameters are:

  • a - the user that is used to connect to the SAS Viya environment.
  • p - the password of the user.
  • sn - the URL of the SAS Viya environment.
  • an - the name of the application that was defined to enable REST APIs.
  • as - the secret to access the REST APIs.
  • rl - the report location should be between quotes if it contains white spaces.
  • rn - the report name should be between quotes if it contains white spaces.
  • o - the output location that will be used.

The output location should ideally be a Git repository. This allows a commit as the next step in the CI/CD process to keep a history log of any report changes.

The output generated by getReport is a JSON file which has the following structure:

{
"name": "CarsReport",
"location": "/Users/sbxxab/My Folder",
"content": {
"@element": "SASReport",
"xmlns": "http://www.sas.com/sasreportmodel/bird-4.2.4",
"label": "CarsReport",
"dateCreated": "2020-01-14T08:10:31Z",
"createdApplicationName": "SAS Visual Analytics 8.5",
"dateModified": "2020-02-17T15:33:13Z",
"lastModifiedApplicationName": "SAS Visual Analytics 8.5",
"createdVersion": "4.2.4",
"createdLocale": "en",
"nextUniqueNameIndex": 94,
 
...
 
}

From the response:

  • The name value is the report name.
  • The location value is the folder location in the SAS Content.
  • The content value is the BIRD representation of the report.

The generated file is not a result of the direct extraction of the report in the SAS environment. It is a combination of multiple elements to build a file containing the required information to import the report in the target environment.

Version the report

The next step in the process is to commit the file within the Git repository. You can use the git commit command followed by a git push to upload the content to a remote repository. Here are some examples:

# Saving the report in the local repository
git commit -m "Save CarsReport"
 
# Pushing the report to a remote repository after a local commit
git push https://gitlab.sas.com/myRepository.git

Promote the report

As soon as you have saved a version of the report, you can import the report in the target environment (production). This is where the postReport comes into play. Here is a sample command line:

python3 postReport.py -a myAdmin -p myAdminPW -sn http://va85.gel.sas.com -an app -as appsecret -i /tmp/CICD/CarsReport.json

The parameters are:

  • a - the user that is used to connect to the SAS Viya environment.
  • p - the password of the user.
  • sn - the URL of the SAS Viya environment.
  • an - the name of the application that was defined to enable REST APIs.
  • as - the secret to access the REST APIs.
  • i - the input JSON file which contains the output of the getReport.py.

The execution of the code returns nothing except in the case of an error.

Testing the report

You now have access to the report in the production environment. A good practice is to test/validate access to the report. While testing manually in an interface is possible, it's best to automate. Sure, you could validate one report, but what if you had twenty? Use the testReport script to verify the report. Below are the command line arguments to execute the code:

python3 testReport.py -a myAdmin -p myAdminPW -sn http://va85.gel.sas.com -an app -as appsecret -i /tmp/CICD/CarsReport.json

The parameters are:

  • a - the user that is used to connect to the SAS Viya environment.
  • p - the password of the user.
  • sn - the URL of the SAS Viya environment.
  • an - the name of the application that was defined to enable REST APIs.
  • as - the secret to access the REST APIs.
  • i - the input JSON file which contains the output of the getReport.py

The testReport connects to SAS Visual Analytics using REST APIs and generates an image of the first section of the report. The image generation process produces an SVG image of little interest. The most compelling part of the test is the duration of the execution. This gives us an indication of the report's performance.

Throughout the CI/CD process, validation is important and it is also interesting to get a benchmark for the report. This is why the testReport populates a .perf file for the report. The response file has the following structure:

{
"name": "CarsReport",
"location": "/Users/sbxxab/My Folder",
"performance": [
{
"testDate": "2020-03-26T08:55:37.296Z",
"duration": 7.571
},
{
"testDate": "2020-03-26T08:55:56.449Z",
"duration": 8.288
}
]
}

From the response:

  • The name value is the report name.
  • The location value is the folder location in the SAS Content.
  • The performance array contains the date time stamp of the test and the time needed to generate the image.

The file updates for each execution of the testReport code.

Conclusion

CI/CD is important to SAS. Many SAS users need solutions to automate their deployment processes for code, reports, models, etc. This is not something to fear because SAS Viya has the tools required to integrate into CI/CD pipelines. As you have seen in this post, we write code to ease the integration. Even if you don’t have CI/CD tools like Jenkins to orchestrate the deployment, you can execute the different Python files to promote content from one environment to another and test the deployed content.

If you want to get more information about ModelOps, I recommend to have a look at this series.

Continuous Integration/Continuous Delivery – Using Python and REST APIs for SAS Visual Analytics reports was published on SAS Users.

4月 102020
 

First introduced in SAS Visual Analytics 8.3, common filters are filters that can be shared between objects in your reports.

Common filter benefits include:

  • Easy to assign the same filter conditions to other report objects.
  • When you edit a common filter, it is updated everywhere that the common filter is used.
  • A common filter is available for the entire report, across pages.

Common filter limitations include:

  • Objects must share the same data source as the common filter definition.
  • Common filters are not available across multiple reports.

While the benefits speak for themselves, common filters also expedite designing reports and exploring data by quickly reusing the same filter conditions with only a few mouse clicks.

Let’s look at some examples of using common filters. In the screenshot below, I defined a filter to return data for the last 30 days. I converted that filter to a common filter and now the Last 30 Days common filter can be applied to any object in this report that uses the same data source. Then I applied it to the bottom treemap object.



In my second example, I parameterized a text input control to capture a user-defined value. Then I defined a parameter-driven filter for the bar chart object. Next, I converted that filter to a common filter named Text Input Contains Filter and applied it to the list table object below.



In my third example, I parameterized two drop-down list controls to capture date boundaries. Then I defined a parameter-driven filter for the waterfall chart object. Next, I converted that filter to a common filter named FromToWeekFilter and applied it to the key value object.



Pro tip: Embrace meaningful filter names. Recall that these common filters are associated with the data source for which they are defined. Being able to quickly identify a common filter definition by its name will save you additional mouse clicks in the long run.

Example 1: Last 30 Days Filter

Let’s pick up where we need to define the filters. Therefore, both objects have been added to the report page with roles assigned.

Next, select the line chart object and open the Filters pane. Click + New filter then select Advanced filter.

Now we will use the built-in filter conditions offered in SAS Visual Analytics. First, select the date/datetime data item, Day. Second, scroll down in the available conditions until you see Last 30 days, then double click on the condition to add it to the expression editor. Third, give the filter condition a meaningful name and as a final step, be sure the number of returned observations is expected. Click OK.

Now we need to convert this filter to a common filter. With the line chart object still selected, on the Filters pane, use the Last 30 Day overflow menu and select Change to common filter.



You should now see the Last 30 Days listed as a Common Filter in the Data pane.

Lastly, we can apply this common filter to the treemap object. Select the treemap object, then open the Filters pane. Next click + New Filter and select the common filter Last 30 Days.



It’s good practice to title your objects to reflect any filters that may be defined for them. Especially if there are no prompts, i.e. control objects, driving the subset of data. This is so your report consumers quickly understand that they are only seeing, in this case, the last 30 days of data.

Example 2: Text Input Contains Filter

This next example will define a parameter-driven common filter. If you are not familiar with using parameters in SAS Visual Analytics, start with this article and refer to the additional materials at the end.

See the screenshot below for more information on the data item role assignments for each object. The most important role that will drive the parameter-driven common filter is the parameter. The parameter, TextInputParameter, is a character parameter assigned to the text input control object and it is the only role assignment. If there is nothing entered in this prompt, i.e. control object, then the filter will return all of the data. We will define a contains expression to only return data rows that contain the entered text.

With the bar chart object selected, open the Filters pane and click + New filter and select Advanced filter.



Now we need to build our parameter-driven filter expression. In this filter we will be checking if the Product Description data item Contains the entered text stored in the TextInputParameter. I wrapped each string expression in an UpCase function so that mixed case is ignored. I could have just as easily used the LowerCase function to get the same result.

And finally, remember to give your filter a meaningful name. The returned observations number is not reflective of an applied filter since I do not have any text entered in the control object. If you had text entered there, then your returned observation number may show a subset of matched rows.



Now we need to convert this object-level filter to a common filter. With the bar chart still the active object, open the Filters pane and use the Text Input Contains Filter overflow menu and select Change to common filter.

You should now see the Text Input Contains Filter in the Data pane.



To add this common filter to the lower list table object, select the list table object and open the Filters pane. Click on the + New filter and select the Text Input Contains Filter.



Now let’s test the filter. In the screenshot below, I’ve typed the word red. Recall that our filter expression is to return rows where Product Description contains the entered text. Notice that even though I do not have the data item Product Description assigned as a data role in the bar chart object, SAS Visual Analytics is still able to apply the filter appropriately. The list table object does have a role assigned for Product Description so that filter application is easier to identify.



I used the contains operator instead of the equals to return a partial match to the Product Description to help identify trends across multiple Products. In these next examples, I entered the text (F) and (M) to return the rows where the Product Description indicates gender-specific products.

Example 3: From – To Week Filter

This last example will also define a parameter-driven common filter. If you need a more step-by-step guide to similar examples, refer to this YouTube video tutorial:

See the screenshot below for more information on the data item role assignments for each object. This example differs from the last in that these control objects, the drop-down lists, use both Category and Parameter Role assignments. The Year-Week data item will provide the available values and the selections will be stored in the parameters FromWeekParameter and ToWeekParameter. I will then create a common filter for an inclusive between of the selected year week values to apply to both the key value object and the waterfall chart.



With the waterfall chart object selected, open the Filters pane and click + New filter and select Advanced filter.



Now we need to build our parameter-driven filter expression. In this filter, I will subset the Year-Week date values which are inclusively between the FromWeekParameter and ToWeekParameter boundaries. Follow these steps:

  1. Select the date/datetime data item, Year-Week.
  2. Scroll down in the available conditions till you see Year-Week BetweenInclusive(‘x’,’y’), then double click on the condition to add it to the expression editor.
  3. Drag the FromWeekParameter and ToWeekParameter from the parameter data items list to the expression.
  4. Give the filter condition a meaningful name. Click OK.



Test the filter by adjusting the values in the drop-down list controls.



Now we need to convert this object-level filter to a common filter. With the waterfall chart still the active object, open the Filters pane and use the FromToWeekFilter overflow menu and select Change to common filter.

You should now see the FromToWeekFilter in the Data pane.



To add this common filter to the key value object; select the key value object and open the Filters pane. Click on the + New filter and select the FromToWeekFilter.

Conclusion

Remember that once a common filter is defined, it can be used for any object in the report that uses the same data source. In these examples, I applied the common filter on the same report page but you can use a common filter on any page within the report. Recall that all of the common filters are listed and available when I click to apply a new filter to an object.

Hopefully these examples have shown you new ways to explore data faster!

Additional materials for using parameters in SAS Visual Analytics:

Using common filters in SAS Visual Analytics was published on SAS Users.

4月 022020
 

Being a state governor has never been more difficult than right at this moment, because the extreme steps governors are being forced to take in order to protect their citizens’ health are having an equally extreme impact on other critical aspects of our society. In the face of social distancing [...]

Governors - here's how your data dashboard can help you lead through the pandemic was published on SAS Voices by Grant Brooks

2月 172020
 

Administrators like to be able to keep track of resource usage and who is using what in a system. When an administrator has this capability, they can look out for issues of high resource usage that may have an impact on overall system performance. In Viya, data is accessed in caslibs. In this post, I will show you how an administrator can track and control resource usage of personal caslibs.

A Caslib is an in-memory space to hold tables, access control lists, and data source information. In GEL enablement classes as we have discussed CAS and caslibs, one of the big asks we have had was related to personal caslibs and how can an administrator keep track of the resources that they use. Until recently we didn’t have a great answer, the good news is now we do.

Personal caslibs are, by default, the active caslib when a user starts a CAS session. This gives each user a default location to access and load data (sort of like a home directory). As the name suggests, they are personal and only the user who starts the session can access the data. In early releases, administrators saw this as a big problem because personal caslibs were basically invisible to them. There was no way to monitor what was going on with a personal caslib, leaving the system open to issues where one user could consume a lot of resources and have an adverse impact.

Introduced in Viya 3.4, the accessControl.accessPersonalCaslibs action brings all existing personal Caslibs into a session where an administrator has assumed the data or superuser role.

Running the accessControl.accessPersonalCaslibs action has the following effects. The administrator can:

  • See all personal caslibs that existed at the time the action was run
  • View promoted tables characteristics within the personal caslibs
  • Drop promoted tables within other users’ personal caslibs.

This elevation of access remains in effect for the duration of the session. The action does not give an administrator full access to personal caslibs, an administrator can never fetch data from other users’ personal caslibs, drop any personal caslib, set access controls on any personal caslib, or set access controls on any table in any personal caslib. What it does is give the administrator a view into personal caslibs to be able to monitor and troubleshooting their resource usage.

Let’s look at how it works. Logged into Viya as Viya administrator (by default also a CAS administrator), I can use the table.caslibinfo action to see all the caslibs that the administrator has permission to view. In the output below, I see my own personal caslib, and all the other caslibs that the administrator has permissions (set by the CAS authorization system) to view.

cas mysess;
proc cas;
table.caslibinfo;
quit;
cas mysess terminate;

In the code below, the super user role is assumed for this session (SAS Administrators by default can also assume the super user role in CAS). With the role assumed, the administrator can execute the accessControl.accessPersonalCaslibs action and the subsequent table.caslibinfo action returns all caslibs including any personal caslibs that existed when the session started.

cas adminsession;
proc cas;
/* need to be a super user or data administrator */
accessControl.assumeRole / adminRole="superuser";
accessControl.accessPersonalCaslibs;
table.caslibinfo ;
quit;
cas adminsession terminate;

That helps, but what about the details? How can the administrator see how many resources the tables in a personal CASLIB are using? To get the details, we can access an individual CASLIB and its tables, and for each table, execute the table.tabledetails action. The program below will loop all the personal caslibs and, for each of the caslibs, it will loop the in-memory tables and execute the table.tabledetails action. The output of tabledetails gives an idea of how many resources (memory, disk etc.) a table is using.

cas adminsession;
proc cas;
/* need to be a super user */
accessControl.assumeRole / adminRole=”superuser”;
accessControl.accessPersonalCaslibs;table.caslibinfo result=fileresult;
casliblist=findtable(fileresult);
 
/* loop caslibs */
do cvalue over casliblist;
 
if cvalue.name==: ‘CASUSER’ then
do; /* only look at caslibs that contain CASUSER */
 
table.tableinfo result=tabresult / caslib=cvalue.name;
 
tablelist=findtable(tabresult);
x=dim(tablelist);
 
if x>1 then
do; /* there are tables available */
 
do tvalue over tablelist; /* loop all tables in the caslib */
 
table.tabledetails / caslib=cvalue.name name=tvalue.name;
table.tableinfo / caslib=cvalue.name name=tvalue.name;
 
end; /* loop all tables in the caslib */
end; /* there are tables available */
end; /* only look at caslibs that contain CASUSER */
end; /* loop caslibs */
 
accessControl.dropRole / adminRole=”superuser”;
quit;
cas adminsession terminate;

Two fields that can give an administrator an idea of how big a table is are:

  • Data size: the size of the SAS Dataset in memory
  • Memory Mapped: the part of the data that has been “memory mapped” and is backed up in the CAS Disk Cache memory-mapped files.

The table below show the output for one users personal caslib.

If one table in particular is causing problems, it is possible for the administrator to drop the table from memory.

cas adminsession;
proc cas;
 
accessControl.assumeRole / adminRole=”superuser”;
accessControl.accessPersonalCaslibs;
sessionProp.setSessOpt / caslib=”CASUSER(gatedemo499)”;
table.droptable / name=”TRAIN”;
 
quit;
cas adminsession terminate;

Visibility into personal caslibs will be a big help to Viya administrators monitoring CAS resource usage. Check out the following for more details:

Viya administrators can now get personal with users' Caslibs was published on SAS Users.

1月 162020
 

Using Customer Lifetime Value in your business decision making is often important and crucial for success. Businesses that are customer-centric often spend thousands of dollars acquiring new customers, “on-boarding” new customers, and retaining those customers. If your business margins are thin, then it can often be months or quarters before you start to turn a profit on a particular customer. Additionally, some business models will segment the worth of their customers into categories that will often give different levels of service to the more “higher worth” customers. The metric most often used for that is called Customer Lifetime Value (CLV). CLV is simply a balance sheet look at the total cost spent versus the total revenue earned over a customer’s projected tenure or “life.”

In this blog, we will focus on how a business analyst can build a functional analytical dashboard for a fictional company that is seeing its revenue, margins, and a customer’s lifetime value decrease and what steps they can take to correct that.

We will cover 3 main areas of interest:

  1. First, screenshots of SAS Visual Analytic reports, using Customer Lifetime Value and how you can replicate them.
  2. Next, we will look at the modeling that we did in the report, with explanations on how we got used the results in subsequent modeling.
  3. Lastly, we talk about one example of how we scored and deployed the model, and how you can do the same.

Throughout this blog, I will also highlight areas where SAS augments our software with artificial intelligence to improve your experience.

1. State of the company

First, we will look at the state of the company using the dashboard and take note of any problems.

Our dashboard shows the revenue of our company over the last two years as well as a forecast for the next 6 months. We see that revenue has been on the decline in recent years and churns have been erratically climbing higher.

Our total annual revenue was 112M last year with just over 5,000 customers churning.

So far this year, our revenue is tracking low and sits at only 88M, but the bad news is that we have already tripled last year's churn total.

If these trends continue, we stand to lose a third of our revenue!

2. The problems

Now, let’s investigate as to where the problems are and what can be done about them.

If we look at our current metrics, we can see some interesting points worth investigating.

The butterfly chart on the right shows movement between customer loyalty tiers within each region of the country with the number of upgrades (on the right) and downgrades (on the left).

The vector plots show us information over multiple dimensions. These show us the difference between two values and the direction it is heading. For example, on the left, we see that Revenue is pointed downward while churns (x axis) are increasing.

The vector plot on the right shows us the change in margin from year to year as well as the customer lifetime value.

What’s interesting here is that there are two arrows that are pointing up, indicating a rise in customer lifetime value. Indeed, if we were to click on the map, we would see that these two regions are the same two that have a net increase in Loyalty Tier.

This leads me to believe that a customer’s tier is predictive of margin. Let’s investigate it further.

3. Automated Analysis

We will use the Automated Analysis feature within Visual Analytics to quickly give us the drivers of CLV.

This screenshot shows an analysis that SAS Visual Analytics(VA) performed for me automatically. I simply told VA which variable I was interested in analyzing and within a matter of seconds, it ran a series of decision trees to produce this summary. This is an example of how SAS is incorporating AI into our software to improve your experience.

Here we can see that loyalty tier is indeed the most important factor in determining projected annual margin (or CLV).

4. Influential driver

Once identified, the important driver will be explored across other dimensions to assess how influential this driver might be.

A cursory exploration of Loyalty Tier indicates that yes, loyalty tier, particularly Tier 5, has a major influence on revenue, order count, repeat orders, and margin.

5. CLV comparison models

We will create two competing models for CLV and compare them.

Here on our modeling page are two models that I’ve created to predict CLV. The first one is a Linear Regression and the second is a Gradient Boosting model. I've used Model Comparison to tell me that the Linear Regression model delivers a more accurate prediction and so I use the output of that model as input into a recommendation engine.

6. Recommendation engine

Based on our model learnings and the output of the model, we are going to build a recommendation engine to help us with determine what to do with each customer.

Represented here, I built a recommendation engine model using the Factorization Machine algorithm.

Once we implement our model, customers are categorized more appropriately and we can see that it has had an impact on revenue and the number of accounts is back on the rise!

Conclusion

Even though Customer Lifetime Value has been around for years, it is still a valuable metric to utilize in modeling and recommendation engines as we have seen. We used it our automated analysis, discovered that it had an impact on revenue, we modeled future values of CLV and then incorporated those results into a recommendation engine that recommended new loyalty tiers for our customers. As a result, we saw positive changes in overall company revenue and churn.

To learn more, please check out these resources:

How to utilize Customer Lifetime Value with SAS Visual Analytics was published on SAS Users.

11月 202019
 

Colleges and universities have access to enormous stores of data and analytics has the power to help higher education tackle some of its biggest challenges. Larry Burns, Assistant Director of Institutional Research and Information Management (IRIM), Oklahoma State University (OSU) knows a great deal about the power of analytics to [...]

Oklahoma State University’s analytics journey was published on SAS Voices by Georgia Mariani

11月 152019
 

Designing interactive reports can be a fun and unique challenge. As user interface experience designers can attest, there are several aspects that go into developing a successful and effective self-service tool. Granted I’m not designing the actual software, but reports require a similar approach to be sure that visualizations are clear and that users can get to the answers they are looking for. Enter prompts.

Reports prompt users to better understand trends, how their data points compare to the whole, and to narrow the scope of data. Being able to pick the placement of these prompts quickly and easily will open the possibilities of your report layouts! I’m specifically speaking about Report and Page level prompts. Traditionally, these global prompt controls were only able to be placed at the top; see the yellow highlighted areas below.

Let’s take a look at an example report with the traditional Report and Page prompt layout. The Report prompts are extremely easy to pick out, since they sit above the pages, but the Page prompts can sometimes blend in with other prompts contained in the report body.

Introduced in the SAS Visual Analytics 8.4 release is the ability to control the layout position of these prompts. Using my example report, let’s change the placement of these prompts. In Edit mode, open the Options pane and use the top level drop-down to select the report name. This will activate the report level, and the report level Options will display. Next, under the Report Controls subgroup, move the placement radio button to the west cardinal point.

Depending on the type of control objects you are using in your report, you may not like this layout yet. For instance, you can see here that my date slider is taking up too much space.

When you activate the slide control, use the Options pane to alter the Slider Direction and Layout. You can even use the Style option to change the font size. You can see that after these modifications, the Report prompt space can be configured to your liking.

Next, let’s change the placement for the Page prompts, for demonstration purposes. From the Options pane, use the top drop-down to select the page name. This will activate the page level, and the page level Options will display. Next, under the Page Controls subgroup, move the placement radio button to the west cardinal position.

You can see that the direction of the button bar control was automatically changed to vertical. Now we can clearly see which prompts belong to the page level.

If I switch to view mode, and adjust the browser size, you can get a better feel for the Report and Page prompt layout changes.

But as with many things, just because you can, doesn’t mean you should. This is where the report designer’s creativity and style can really take flight. Here is the same report, but with my preferred styling.

Notice that I kept the Report prompts along the top but moved the Page prompts to the left of the report. I also added two containers and configured a gray border for each container to better separate the objects. This helps the user quickly see that the drop-down will filter the word cloud is only. I also used the yellow highlighting through styling and a display rule to emphasize the selected continent. The bar chart is fed from an aggregated data source which is why the report prompt is not filtering out the other continents.

Feel free to send me your latest report design ideas!

Additional material related to Report and Page prompts:

New control prompt placement option in SAS Visual Analytics was published on SAS Users.

7月 192019
 

When a new Moon passes between the Earth and the Sun, the Moon can cast a shadow on certain regions of the Earth. This natural phenomenon creates a solar eclipse, meaning the Moon covers, or eclipses, your view of the Sun if you're in that region. No surprise that in [...]

Ring of fire: Visualizing 5,000 years of solar eclipses was published on SAS Voices by Falko Schulz

4月 052019
 

When working with files like SAS programs, images, documents, logs, etc., we are used to accessing them in operating system directories. In Viya, many of these files are not stored on the file-system. In this blog, we will look at where and how files are stored in Viya, and how to manage them.

In Viya, external files are stored in the Infrastructure Data server and accessed using the file micro-service. The file service manages a wide variety of file types including, images, html files, text files, csv files, SAS programs, media files, pdfs, office documents and logs. The file service is not a complete file system, but rather a method of accessing individual files stored in the infrastructure data server identified via their URI (a unique identifier).

Some files may also be accessible in Viya folders. The file service stores resources like images, SAS programs, etc. in the Infrastructure Data server, but they are accessible from Viya folders in the Content Area of Environment Manager or in SAS Drive.

The screenshot below shows the properties of an image file that was uploaded to a Viya folder using SAS Drive. You can see the filename, the URI (including the ID) and the location in the folder structure.

For files that are surfaced in folders, we can manage the files (view, copy, move, delete etc.) using SAS Environment Manager Content area, SAS Drive or the Folders command line interface. To learn more, watch this video from the SAS Technical Insights and Expertise Series.

However, many system generated files are managed by the file service and stored in the infrastructure data server, but are not accessible from the folders interfaces. A good example of system generated files that an administrator may need to manage are logs.

Many times when processing in Viya, logs are created and stored in the infrastructure data server. For example, when CAS table state management jobs, data plans, file imports, model or scoring processes are executed, a log is generated and stored by the file service.
How do we manage these files? While the files generated are mostly small in size, a large active Viya system with a long history will need management of the log and other files stored in the infrastructure data server.

One way we can access these files is using the REST API of the file service.

You can use your favorite tool to access the REST API. In this blog, I will use the GEL pyviyatools, initially the basic callrestapi tool. For more details on the pyviyatools, read this blog or go directly to the SAS GitHub site.

To get a list of all files are stored in the infrastructure data server send a GET request to the /files/files endpoint of the files service.

/./callrestapi.py -m get -e /files/files -o simple/

Partial output shows the image file referenced above and what could be a system generated log file.
=====Item 0 =======
contentType = image/jpeg
createdBy = geladm
creationTimeStamp = 2019-01-24T14:40:15.085Z
description = None
encoding = UTF-8
id = 6d2995b3-0fa6-4090-a338-1fbfe51fb26b
modifiedBy = geladm
modifiedTimeStamp = 2019-01-25T17:59:28.500Z
name = company_logo.JPG
properties = {}
size = 327848
version = 2

=====Item 164 =======
contentType = text/plain
createdBy = sasadm
creationTimeStamp = 2019-01-31T12:35:00.902Z
description = None
encoding = UTF-8
id = e4d4c83d-8677-433a-a8d3-b03bf00a5768
modifiedBy = sasadm
modifiedTimeStamp = 2019-01-31T12:35:09.342Z
name = 2019-01-31T12:35.823Z-results.log
parentUri = /jobExecution/jobs/380d3a0c-1c31-4ada-bf8d-4db66e786669
properties = {}
size = 2377
version = 2

To see the content of a file, use a GET request, the id of the file and the content endpoint. The content of the log file is displayed in the call below.

/./callrestapi.py -m get -e /files/files/01eb020f-468a-49df-a05a-34c6f834bfb6/content/

This will display the file contents. The output shows the log from a CAS table state management job.

———-JOB INFORMATION———-

Job Created: 2019-01-31T12:35.823Z
Job ID: 380d3a0c-1c31-4ada-bf8d-4db66e786669
Heartbeat interval: PT0.3S
Job expires after: PT168H
Running as: sasadm
Log file: /files/files/e4d4c83d-8677-433a-a8d3-b03bf00a5768 [ 2019-01-31T12:35.823Z-results.log ]
Arguments:

Options:
{
“enabled” : true,
“type” : “LOAD”,
“settings” : {
“refresh” : false,
“refreshMode” : “newer”,
“refreshAccessThreshold” : 0,
“varChars” : false,
“getNames” : true,
“allowTruncation” : true,
“charMultiplier” : 2,
“stripBlanks” : false,
“guessRows” : 200,
“scope” : “global”,
“encoding” : “utf-8”,
“delimiter” : “,”,
“successJobId” : “”
},
“selectors” : [ {
“serverName” : “cas-shared-default”,
“inputCaslib” : “hrdl”,
“outputCaslib” : “hrdl”,
“filter” : “or(endsWith(tableReference.sourceTableName,’.sashdat’), endsWith(tableReference.sourceTableName,’.SASHDAT’),\nendsWith(tableReference.sourceTableName,’.sas7bdat’),\nendsWith(tableReference.sourceTableName,’.csv’)\n)”,
“settings” : { }
} ]
}

————————————————–

Created session cas-shared-default: d4d1235a-6219-4649-940c-e67526be31ed (CAS Manager:Thu Jan 31 07:35:01 2019)
Using session d4d1235a-6219-4649-940c-e67526be31ed
Server: cas-shared-default
Input caslib: hrdl
Output caslib: hrdl
Effective Settings:

———-LOAD STARTING———-

— loaded –> unloaded – HR_SUMMARY unloaded – HR_SUMMARY_NEW unloaded – HRDATA unloaded – PERFORMANCE_LOOKUP <– performance_lookup.sas7bdat
Access denied.
———-LOAD COMPLETE———-

———-CLEANUP STARTING———-

Session deleted: cas-shared-default: d4d1235a-6219-4649-940c-e67526be31ed
———-CLEANUP COMPLETE———-

Final Job State: completed
Log file:/files/files/e4d4c83d-8677-433a-a8d3-b03bf00a5768

I started my journey into the file service to discover how an administrator could manage the log files. In reviewing the files and their contents using the REST API, I discovered that there was no easy way to uniquely identify a log file. The table below shows the attributes of some of log files from the GEL Shared Viya environment.

As this post has illustrated, file service REST API can be used to list and view the files. It can also be used for other file management activities. To help administrators manage files that are not visible via a user interface, a couple of new tools have been added to the pyiyatools.

listfiles.py provides an easy interface to query what files are currently stored in the infrastructure data server. You can sort files by size or modified date, and query based on date modified, user who last modified the file, parentUri or filename. The output provides the size of each file so that you can check the space being used to store files. Use this tool to view files managed by the file service and stored in the infrastructure data server. You can use lisfiles.py -h to see the parameters.

For example, if I want to see all potential log files older than 6 days old created by the /jobexecution service, I would use:

/./listfiles.py -n log -p /jobExecution -d 6 -o csv/

The output is a list of files in csv format:

id ,name ,contentType ,documentType ,createdBy ,modifiedTimeStamp ,size ,parentUri

“f9b11468-4417-4944-8619-e9ea9cd3fab8″,”2019-01-25T13:35.904Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T13:35:09.490Z”,”2459″,”/jobExecution/jobs/37536453-fe2f-41c2-ba63-ce72737e482c”

“dffdcc97-3fb0-47c1-a47d-e6c8f24f8077″,”2019-01-25T12:35.869Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T12:35:08.739Z”,”2459″,”/jobExecution/jobs/99708443-2449-40b7-acc3-c313c5dbca23″

“5fa889b6-93fb-4496-98ba-e0c055ca5999″,”2019-01-25T11:35.675Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T11:35:09.118Z”,”2459″,”/jobExecution/jobs/eb182f88-4853-41f4-be24-225176991e8a”

“87988659-3c2d-4602-b61a-8042b34022ac”,”2019-01-25T10:35.657Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T10:35:09.881Z”,”2459″,”/jobExecution/jobs/73fffe47-a7ef-4c1d-b7bf-83ef0b86319e”

archivefiles.py allows you to read files from the file service and save them to a directory on the file system. Optionally, the tool will also delete files from the file service to free up space. For example, if I want to archive all the files I listed above, I would use:

/./archivefiles.py -n log -d 6 -p /job -fp /tmp/

This tool will create a timestamp directory under /tmp and save a copy of each file in the directory.

If you want to archive and delete, add the -x option.

IMPORTANT: Use the archive tool carefully. We recommend that you run a Viya Backup prior to running the tool to delete files.

Now you know where your files are and you have some help with managing them. For the full details of how you can manage files using the file service REST API you can view the file service REST API documentation on developer.sas.com. If you would like to suggest any changes to the existing tools, please enter a suggestion on GitHub.

Where are my Viya files? was published on SAS Users.