Tech

5月 202016
 

ProblemSolversPROC FREQ is often the first choice when you want to generate basic frequency counts, but it is the last choice when it is compared to other statistical reporting procedures. People sometimes consider PROC FREQ last because they think they have little or no control over the appearance of the output. For example, PROC FREQ does not allow style options within the syntax, which the REPORT and TABULATE procedures do allow. Also, you cannot control the formats or headings with statements in the procedure step.

Sometimes, a simple frequency (via a one-way table) is all you want, and you don’t want to have to create an output data set just to add a format. A one-way table in PROC FREQ is unique also in that it includes a cumulative count and percent. These calculations cannot be done in other procedures without additional code or steps. However, there is a simple way to make some basic modifications to your table. By adding a PROC TEMPLATE step to modify either the Base.Freq.OneWayFreqs or the Base.Freq.OneWayList table template, you can change the formats of the statistics, move the label of the variable, change the labels of the statistics, and suppress the Frequency Missing row that appears below the table. These changes apply to all output destinations, including the traditional listing output.  You can also use PROC TEMPLATE to make small modifications to the nodes that are generated by one-way tables in the table of contents for non-listing ODS destinations.

Customize the Formats and Labels for Statistics

If you want to modify the format for the statistics Frequency, Cumulative Frequency, Percent, and Cumulative Percent, you can use the FORMAT= option in PROC TEMPLATE. This option accepts SAS formats and user-defined formats. If you want to change the statistic heading, you can use the HEADER= option.

The following example uses both the FORMAT= and the HEADER= statements to format the statistics values and to change the Frequency heading to Count. This example also changes the Percent heading to Pct.

proc format;
     picture pctfmt (round) other='009.9%';
run;

proc template; 
     edit Base.Freq.OneWayList;
        edit frequency;  
           header='Count'; 
           format=comma12.;
        end;  

        edit cumfrequency;
           format=comma12.; 
        end;
edit percent;
header='Pct';
format=pctfmt.;
end;
   
edit cumpercent;
format=pctfmt.; 
end; 
end;
run;                                                                                  

data class;
     set sashelp.class;
     wgt=1000;
run;
 
proc freq data=class;
     tables age;
     weight wgt; 
run;

This code generates the following table:

FREQ procedure's one-way tables01

Move the Variable Label

If a variable has a label, it is centered, by default, at the top of the table, and the variable name appears above the column of values.

FREQ procedure's one-way tables02

If you want to use the label instead of the name above the column, you can edit the HEADER= value in PROC TEMPLATE, as shown in the following example:

proc template;
   edit Base.Freq.OneWayList;             
        edit h1; 
                /* Set text to a blank instead of VARLABEL. */ 
             text " ";
        end;                                                
                                                                                                  
        edit FVariable;                                           
           just=varjust;
           style=rowheader;
           id;
           generic;
           header=varlabel;
        end;
     end;
  run;

proc freq data=sashelp.class;
   label age='Age in Years';
   tables age;
run;

This code generates the following table, which replaces the default location of the label with blank text and moves the label so that it is above the column of the variable's values, as shown in this example:

FREQ procedure's one-way tables03

Suppress the Frequency Missing= Row

If a variable has missing values and you do not include the MISSING option in the TABLES statement of PROC FREQ, the output includes the frequency of missing values (Frequency Missing=) below the table with the number of missing values.

The following table shows the default output:

FREQ procedure's one-way tables04

To suppress this line without including missing values in the table output, you can modify the table template, as follows:

data test;
input c1 $;
cards;
a
b
.
;
run;

proc template;
     edit Base.Freq.OneWayFreqs; 

   /* This is the default setting:                     */ 
   /*  define f2;                                      */
   /*     text "Frequency Missing =" fMissing -12.99; */
   /*     print = pfoot2;                              */
   /*  end;                                            */
        edit f2; 
           text " ";
        end;
     end;
run;

proc freq data=test;
   table c1;
run;

The table that is generated by this code does not contain the Frequency Missing= row at the bottom of the table.

FREQ procedure's one-way tables05

Customize the Table of Contents

For a one-way table, the default table of contents looks like this:

FREQ procedure's one-way tables06

To control the first node, The Freq Procedure, you can use the ODS PROCLABEL statement.  You can change the text of this node, but you cannot eliminate the node. You can remove the One-Way Frequencies node using PROC TEMPLATE with the CONTENTS= statement. The following example changes the text of the The Freq Procedure node, and it eliminates the One-Way Frequencies node.

proc template;
   edit Base.Freq.OneWayFreqs;
      contents=off;
   end;
run;

ods listing close;
ods pdf file='test.pdf';

ods proclabel='My One-Way Table';
proc freq data=sashelp.class;
tables age;
run;

ods pdf close;
ods listing;

Here is the modified table of contents:

FREQ procedure's one-way tables07

(For additional customization of the table of contents, you need to consider using the DOCUMENT procedure.  The SAS Global Forum 2011 paper Let's Give 'Em Something to TOC about: Transforming the Table of Contents of Your PDF File, by Bari Lawhorn, contains useful information and examples that illustrate how to use PROC DOCUMENT, PROC TEMPLATE, and other SAS statements and options to customize your TOC.)

To restore the default table template, run the following code:

proc template;
delete Base.Freq.OneWayFreqs;
delete Base.Freq.OneWayList;
run;

The examples above show how you actually can customize PROC FREQ one-way tables, with a little help from PROC TEMPLATE. PROC TEMPLATE offers several methods that make it easy to customize one-way, frequency-count tables that enhance table format and appearance.
tags: Problem Solvers, PROC FREQ, SAS Programmers

Seeing the FREQ procedure's one-way tables in a new light was published on SAS Users.

5月 182016
 

EnvironmentManagerMy previous blog discussed the SAS Environment Manager Report Center, and talked about its organization, and how to start using some of the prompts to get the reports you want. The next step is to learn to use some of the example reports provided to help you design your own, production-level reports.

It’s helpful to distinguish ad-hoc, or exploratory reporting, vs. standardized, production-style reports, which might be run daily, weekly, or any other number of times. You can experiment with many ad-hoc reports and create hundreds of variations and different reports “on the fly,” just by manipulating the various parameters provided within the Report Center interface. Perhaps more importantly, you can use knowledge gained from this process to design more permanent production reports, using some of the provided examples as “templates.”

First, you need to understand the basic structure: All the SAS Environment Manager Reports are SAS Stored Processes available from the Stored Process web application. Most of them accept user prompts that specify many details about the data to include and the way the output should appear. Then, the stored process passes these parameters to a small set of reporting macros, which perform the data manipulations and produce the actual report using well-known SAS Procedures.

In addition to all the pre-defined reports, there are two sets of example reports specifically intended to be used as templates from which you can develop your own custom report. Often, this only requires a few changes to parameters being passed to the reporting macro. These example reports are located here:

Under Products>SAS Environment Manager>Custom
Under System>Applications>SAS Environment Manager>Ad Hoc Reporting

Creating custom reports from the Data Mart01

In the first set, the Example2 Call EV Macros with prompts is likely of most interest. This report demonstrates how to call two of the most useful reporting macros:

%evtmpplate creates a wide variety of plots using PROC TEMPLATE
%evtabrpt  creates a wide variety of data tables using PROC TABULATE

The second set in Ad Hoc Reporting has a series of reports that use many additional reporting macros. Notice that many of the reports include the reporting macro name they showcase, which matches the corresponding stored process filename, in the report name itself. Thus you can quickly locate the most important files being used in the running of the report based on the report’s name. Here’s a table showing the most useful reporting macros:

Creating custom reports from the Data Mart02

Armed with this information, here’s the approach to take to create your own custom reports in the Report Center:

  1. Decide what you want to report on, and what general type of report you want to create (table, crosstab, bar chart, line graph, heat map, etc.)
  2. Try to find existing reports in the Report Center that may already do what you want, or are at least based on the data that you want to use
  3. Determine which Data Mart data set you want to report from, based on Steps 1 and 2
  4. If none of the existing reports provide what you want, try experimenting with all the parameters available in the Ad-Hoc reports section, (ACM, ARTIFACT, KITS libraries only) using the report type you want (plot, chart, tabulate, etc.) to determine the example report to try.  Most of these reports allow you a wide variety of choices, including:
    o  data set
    o  main analysis variables
    o  class variables, or BY variables
    o  type of plot, plus many options and details about the plots
    o  data subsetting by variable (WHERE clause), dates
    o  output formatting
    o  titles/footnotes/ODS style, etc.
  5. When running the Ad Hoc reports, first specify the data set and analysis variable(s) that you want, and then experiment with various other parameters. Open the file containing the macro code that you are using–the reporting macros are fully documented with all options and parameters accepted. Use the table above to help you find the correct file(s).
  6. Once you get the report that you want from the Ad Hoc section, then you can use the information you’ve gained to create a more permanent version. Here’s an overview of the steps to create a permanent custom report:

Caution before proceeding: Be careful not to modify any of the stored process files that are part of the Report Center. If you want to modify one of the stored processes, first make a copy and give it a new name. Then simply refer to the new name in that stored process’ metadata and modify the copy.  Or, you can put the stored process code directly in metadata instead of a file, in which case you don’t have to alter any Report Center files.

  1. Using SAS MC, copy the existing Ad Hoc report to a new metadata location, and rename it.
  2. Make a copy of the stored process file called by the report, and rename it.
  3. Edit the new (copied, renamed) stored process code, and you will see the reporting macro(s) being called.
  4. Modify the parameters on any called reporting macros with the new parameters you want, based on what you learned in the previous steps.  For more guidance, open the *.sas file containing the reporting macro, and follow the detailed documentation that’s included with each macro.
  5. Chances are that it will require a few tries before you get it exactly right, but it’s nice to see how much of the work has already been done for you with the existing examples.

More information on the steps to add a report to Report Center.

 

tags: SAS Administrators, SAS Environment Manager, SAS Professional Services

Creating custom reports from the Data Mart was published on SAS Users.

5月 162016
 

Highlights from SAS Global Forum 2016SAS Global Forum 2016 recently concluded and by all measures it was a huge success. With more than 5,000 attendees, hundreds of crazy-good talks and countless networking and learning opportunities, it very likely was the best SAS users’ conference yet. To put a bow on this year’s event, and help those who were unable to attend enjoy some of the great content, I’ve put together a few highlights from SAS Global Forum 2016, with some helpful links, videos and several additional resources to explore.

Before we get started, though, I wanted to share a big thank you to all the volunteers who made SAS Global Forum 2016 possible. You don’t become the world’s largest analytics conference without substantial support from the user community and an outstanding roster of hard-working and dedicated volunteers.

This year, more than 450 users, led by conference chair Jenn Waller, volunteered countless hours of time and tons of energy to make SAS Global Forum such a success. Within this amazing group, we wanted to highlight the inaugural Global Forum Service Award recipients. These individuals were recognized for their dedication and commitment, not only to this year’s event, but for having served at least five years as a team lead or a section chair at SAS Global Forum. The award recipients are:

  • Maribeth Johnson – 12 years
  • Diana Suhr – 12 years
  • Nancy Brucken – 11 years
  • Pete Lund – 11 years
  • Sue Douglass – 9 years
  • Mary Anne DePesquo – 8 years
  • Christianna Williams – 8 years
  • Stephanie Thompson – 5 years

To learn more about the benefits of volunteering, read SAS Global Forum 2016 Chair Jenn Waller’s blog, Can you help? Volunteer opportunities aplenty at SAS Global Forum 2016. There are a number of opportunities for you to get involved in next year’s conference or one of SAS’ Regional User Group events this fall.

Now for the recap. Here are some highlights from SAS Global Forum and some helpful links and resources in case you weren’t able to attend.

Exciting announcements

The conference began with a bang. At Opening Session, Dr. Goodnight introduced several new offerings from SAS. Perhaps the most exciting was the announcement of SAS Viya, the next-generation high-performance analytics and visualization architecture from SAS. We also introduced other new products, including SAS Analytics for IoT and SAS Customer Intelligence 360.

Check out archived video of SAS Global Forum 2016 Opening Session.

Outstanding speakers

There were hundreds of top-notch session and keynote talks, but there were two in particular that really resonated with me.

The first was Arianna Huffington’s talk, Redefining Success: The Third Metric That Can Benefit Your Bottom Line. In her presentation, the co-founder, president and editor-in-chief of the Huffington Post Media Group, refuted the popular belief that in order to be successful we need to burn ourselves out. Instead, she challenged the audience to take care of themselves and their relationships and watch success follow.

Susan Cain’s keynote presentation was another one that hit the mark for me. The author of Quiet: The Power of Introverts in a World That Can’t Stop Talking, Cain discussed how extroverts can dominate the workplace simply because of their predisposition to speak up, not because of their great ideas. (Yes, sometimes a problem I have!) She had some great advice for how introverts and extroverts could work together more effectively, resolve conflict and how to benefit from each other’s strengths.

View Huffington’s talk in its entirety

View Cain’s talk in its entirety

View other keynote and select breakout presentations on the SAS Video portal

Great networking

For attendees, SAS Global Forum offered a number of awesome opportunities to interact with and learn from other SAS professionals. SAS Global Forum was covered extensively on SAS Blogs (aggregated here), the SAS Support Communities, the SAS Twitter Account (hashtag #SASGF), SAS Users LinkedIn group, and the SAS Users Group Facebook page. But, building a broader network of SAS peers, doesn’t have to end with the conclusion of the event. Whether you attended the conference or not, connections with other SAS users can continue via social media. The channels mentioned above are great follows if you’re looking to get the latest scoop from SAS, start a conversation with your SAS peers, or share tips and tricks.

You might also want to check out this fun blog: Ten SAS Global Forum speakers to follow on Twitter. Now there’s an awesome network for you to follow!

Valuable papers

If you’re a SAS user, there’s nothing better than SAS Global Forum presentations. This year had a record number of presentations from users and SAS subject matter experts. Even if you attended the event, you only got a taste of all that great content. The good news: conference papers (and more) are available on the SAS Global Forum Proceedings webpage. Here you can view all papers or filter by topic, session type, skill level, job role, industry, or product. Heck, you can also download all proceedings if you’d like!

Extensive collection of archived video

During the event, SAS provided dozens of hours of live video streaming. Live coverage included Opening Session, all the keynote talks, select breakouts, Tech Talks, interviews with SAS executives and developers, and so much more. If you didn’t go to SAS Global Forum, or did but missed a few sessions, you can view some of the great content via SAS’ live stream archive.

So there you have it. A few highlights from SAS Global Forum 2016. If you attended SAS Global Forum this year, thank you. If you didn’t make it to Las Vegas, enjoy these resources and we hope you can join us next year in Orlando!

tags: SAS Global Forum

Couldn’t attend SAS Global Forum 2016? Here are a few highlights, resources and thank-yous was published on SAS Users.

5月 142016
 

Trusted data is key to driving accurate reporting and analysis, and ultimately, making the right decision. SAS Data Quality and SAS Data Management are two offerings that help create a trusted, blended view of your data. Both contain DataFlux Data Management Studio, a key component in profiling, enriching monitoring, governing and cleansing your data. Clustering, or grouping similar names or addresses together, is one data quality activity that helps to reduce the number of duplicates in a given data set. You do this by using fuzzy matching to group similar names or addresses together. As an example, a marketing analyst might want to remove duplicate customer names or addresses from a customer list in order to reduce mailing costs.

Have you ever wondered how the cluster results would differ if you changed the match code sensitivity for one of your data columns, or removed a column from one of your cluster conditions or added a new cluster condition? Well, wonder no more!

The Cluster Diff node in DataFlux Data Management Studio compares the results of two different Clustering nodes based on the same input data.  This is useful for comparing the results of different cluster conditions and/or different match code sensitivities.

All records from the input set must be passed to both Clustering nodes and both Clustering nodes must pass out all their data in the same order for this comparison to work. To summarize, in both Clustering nodes, you must select the All clusters output option and you cannot use the Sort output by cluster number option.

DataFlux Data Management Studio with Cluster Comparison

Clustering Node Properties

The results of both Clustering nodes are then fed into the Cluster Diff node.  In order to perform the comparison, the Cluster Diff node must know the unique identifier for the input records (Record ID) and the Cluster number that is returned from the respective Clustering node.

DataFlux Data Management Studio with Cluster Comparison

Cluster Diff Node Properties

The Diff type value describes the type of change when performing the cluster number comparison between the two Clustering nodes. The possible values for Diff type include COMBINE, DIVIDE, NETWORK, and SAME which is represented as a period (.). When comparing the results of the two Clustering nodes the results are reviewed as a Diff set.  Within a Diff set:

  • If the records were in different clusters on the “left table” and in the same cluster on the “right table,” then its Diff type is COMBINE.
  • If the records were in the same cluster on the “left table” and in different clusters on the “right table,” then its Diff type is DIVIDE.
  • If the records were in same cluster on the “left table” and in the same cluster on the “right table,” then its Diff type is “.” (SAME).
  • If when comparing the “left table” cluster to the “right table” clusters at least one record is added to the cluster AND at least one record is removed from the cluster, then its Diff type is NETWORK.
DataFlux Data Management Studio with Cluster Comparison

Cluster Diff Node Results

The Cluster Diff node is not a node that is typically used in a production matching job. However, it is a node that is useful in helping you compare and contrast different match code sensitivities and/or cluster conditions that enable you to achieve the best matching results for your data set.

 

tags: data management, DataFlux Data Management Studio, SAS Professional Services

Improving matching results in DataFlux Data Management Studio with cluster comparison was published on SAS Users.

5月 122016
 

SASMobile_BI_FAQSAS Visual Analytics users view and interact with reports on their desktop computers or laptops. Many, however, have never heard of the SAS Mobile BI app or how it extends the viewing and interactive capabilities of their reports to mobile devices. SAS Mobile BI app is simple to download to a mobile device, and you can immediately view some sample SAS Visual Analytics reports from within the app.

If your organization has deployed SAS Visual Analytics, but is not taking advantage of the ability to view and interact with reports via mobile devices, I urge you to consider it.  Almost every type of interaction that you have with a SAS Visual Analytics report on your desktop – you can do the same with reports viewed in SAS Mobile BI!

It’s worth noting that having SAS Visual Analytics in your organization is not a requirement for downloading this nifty, free app on to your Android or iOS mobile devices.  Once you download the app, you can view and interact with a wide spectrum of sample SAS Visual Analytics reports for different industries.

Like what you see?

If so, talk with the SAS Visual Analytics administrator in your organization and ask them to enable support for viewing your SAS Visual Analytics reports in the SAS Mobile BI app.

To give you a little more guidance, here are some of the most frequently asked questions about SAS Mobile BI.

Do we need a special license to use SAS Mobile BI?

Absolutely not! It is a free app from SAS that is available to anyone who wants to download it on to an Apple or Android device and view reports and dashboards.

Do we need to have SAS Visual Analytics in our organization to use the app?

No, you don’t.

When the app is downloaded to your device, sample charts and reports are made available to you by SAS via a server connection to the SAS Demo Server.

How do we get this free app?

You can download the free app to your Apple or Android device from:

Apple iTunes Store
Google Play

After downloading the app, what are some things we can do with it?

Well, as soon as you download the app and open it, you have a connection established to the SAS Demo Server which hosts a variety of sample reports with data associated with different industries. These reports give you an idea of the various features that are supported by the app. Here is a snapshot of the different folders that contain sample reports for you to browse and view.

 

SASMobileBIFAQ1

Here is the Portfolio where subscribed reports are available in the form of thumbnails that can be tapped to open them.

SASMobileBIFAQ2

If we have reports in our organization that were created with SAS Visual Analytics Designer, can we view those reports in this app?

Absolutely, yes!

The same reports that you view in your web browser on a desktop can be viewed in SAS Mobile BI by using your Apple or Android devices (both tablets and smartphones).  Simply define a connection to your company server and browse for reports. Access to reports from SAS Mobile BI is granted by your SAS Administrator. Report access can be based upon various Capabilities and Permissions.

Live data access requires either a Data or a Wi-Fi connection, and your company may require VPN access or other company-specific security measures. Contact your SAS Visual Analytics Administrator. Also, be sure to mention that there is a comprehensive Help menu within the app that provides answers to queries they might have.

What type of features are available in reports that are viewed from SAS Mobile BI?

Typically, almost all of the features that are available in the desktop-based reports are supported in the app. Drilling, filtering, brushing, display rules, linking, info windows – are just some of the key features available to you when you interact with reports via this app.

I hope you’ve found this blog and the accompanying FAQ helpful. In my next blog I’ll cover some basic security features we’ve built into the app to protect user data.

tags: SAS Mobile BI, SAS R&D, SAS Visual Analytics

SAS Mobile BI FAQ was published on SAS Users.

5月 112016
 

SASViyaI’ve been giving presentations about SAS® ViyaTM for a couple of months now, and the reactions have been positive. I’m part of a much bigger cast of speakers at SAS who talk about the new analytics platform with key customers and analysts. While some presenters focus on the overarching benefits or the integration with previous versions of SAS, I get to focus on hands-on demonstrations. My style is to show SAS Viya technology in use, which helps attendees understand the different user experiences.

The demonstrations I show use anywhere from 90,000 to 2.5 million observations. Sometimes I’m looking for fraud or sometimes I’m looking for opportunities to reduce attrition. No matter what business problem I’m trying to solve, I like to show four different ways to tackle the problem, depending on the user persona and skillsets.

First, I show SAS® Visual Statistics. This is a suitable interface for business analysts and citizen data scientists. I can point and click to do a logistic regression and find an answer. Or, I can start to explore my data with SAS® Visual Analytics before I do any modeling.

Running a logistic regression in SAS Visual Statistics

Running a logistic regression in SAS Visual Analytics (click to enlarge).

Next, I show SAS® Studio, where you can also point and click, or you can program SAS code. You can do both here, or toggle back and forth, which provides a lot of benefits when reviewing how your code works. Here, I run a logistic regression, write the procedure and use PROC LOGSELECT, which is the logistic procedure in SAS Viya. I also point out all of the exploratory and descriptive tasks and procedures you can use before fitting your model.

StudioLogistic (002)

SAS Studio shows the selected model and the code side by side (click to enlarge).

Then, I tab over to Jupyter Notebook and show the CAS Python API. (CAS is short for cloud analytic server.) I can write Python code that calls specific CAS actions, like calling the logistic action set or performing model assessment. The CAS Python API is not yet released but will be available later this year.

SAS code in Python

Performing a model assessment from inside Jupyter Notebook.

Finally, I like to show these prepackaged predictive models that run SAS through APIs. Using analytics as a service by SAS, these APIs can be used to embed logistic regression or other modeling techniques into any other application.

A3s (002)

APIs are portable pieces of code that can be easily combined and stacked together to enhance other applications or websites (click to enlarge).

I use the same data for all of the examples and because the same CAS action set is used for analysis you get the same results. For each one, SAS Viya goes right to the data and lifts it up into memory. I can perform my analyses quickly and nimbly, and when I’m done, the data dribbles back down to its initial location instead of proliferating copies of the data on hard disks.

Not only is it quick and easy to get complex answers – but you can get them in whatever way feels most comfortable to you.

Which of the four do you prefer? And what ideas do the different options inspire in you?

tags: Jupyter Notebook, Python, SAS Viya

Choose your own adventure with SAS Viya was published on SAS Users.

5月 042016
 

New default parameter values for Platform Suite for SASNew default parameter values for Platform Suite for SAS

Sometimes, when your kids grow older, they change their habits and you don’t recognize their behaviors any more. “We play this game every year at the beach. Don’t you like it anymore?” you ask. “Dad, I’m not seven years old any more”.

Well, Platform Suite for SAS is not seven any more. And its default behavior has changed, too.

Recognizing the release

Platform Suite for SAS ships with SAS Grid Manager offering, and (almost) every SAS maintenance changes the bundled release. It includes different products that do not share the same numbering sequence.  we are currently (as of 9.4M2) shipping Platform Suite for SAS version 8.1, which includes LSF 9.1.1.

Any new release adds additional features, expands the list of supported operating systems and increases the flexibility in configuring your environments. But the default values of the main parameters, that characterize how the software behaves out of the box, are usually untouched. Until now.

A faster start

After installing a new environment, we suggest to submitting many jobs, all at once, to check that LSF dispatches them correctly. If you have ever done this more than once, you surely remember that jobs take a while to start. The following screenshot shows the results of a bhist command issued about a minute after submitting 15 jobs on a newly installed, 3-nodes-grid that uses LSF 8.01 (SAS 9.4 and 9.4M1).

15 jobs submitted to a LSF 8 grid

15 jobs submitted to a LSF 8 grid

You can see that jobs are kept pending (column highlighted in red) while LSF decides which host is the best to run them. LSF starts jobs in subsequent batches every 20 seconds, and after a minute some jobs have not started yet (column highlighted in green).

Here is another screenshot, that shows the same bhist command issued only thirty seconds after submitting 15 jobs on a newly installed, 3-nodes-grid that uses LSF 9.11 (SAS 9.4M2). Can you spot the difference? All jobs start almost immediately, without losing any time in the pending state:

15 jobs submitted to a LSF 9 grid

15 jobs submitted to a LSF 9 grid

Isn’t this good?

Well, it depends. Platform LSF is a system built and tuned for batch interaction. As such, many components need some “think time” before being able to react. At the end, when submitting a 2-hour-long batch job, is it important if it takes 1 or 25 seconds to start?

Things change when we use LSF to manage interactive SAS workloads. End users do care if a SAS session takes 5 or 25 seconds to start when submitting a project from SAS Enterprise Guide. If it takes more than 60 seconds, the object spawner may even time out. The practice we use is to tune some LSF parameters, as shown in this blog post, to reduce grid services sleep times so that interactive sessions start faster.

Looking at the above results, how fast jobs start with LSF 9.1.1, gives us a hint that the new release has these parameters already tuned by default. That’s good! Isn’t it?

Comparing the values

To understand which parameters have new default values, it is possible to compare the Platform LSF Configuration Reference Version 9.1 with Platform LSF Configuration Reference Version 8.01. I verified what I found there by checking the actual LSF configuration files in two grid installations, then I built the following table to compare the main parameters we usually tune:

New default parameter values for Platform Suite for SAS3

The “Default” column reports values that are automatically set in configuration files after a default deployment. When a parameter is not defined in the configuration files, it takes the value listed under the “Undefined” column. As you can see, all of the actual values have been lowered.

What does this mean?

With the new default LSF 9 values, a SAS 9.4M2 grid is more responsive to interactive users and can accept more jobs that are submitted all at once, increasing the overall job throughput. Grid-launched workspace servers now start almost immediately (if there are enough resources to run them, obviously) with no timeouts or long waiting.

… BUT …

There is one problem with this configuration.

If you are familiar with LSF tuning, you may remember this note from the official documentation:

JOB_ACCEPT_INTERVAL: If 0 (zero), a host may accept more than one job. By default, there is no limit to the total number of jobs that can run on a host, so if this parameter is set to 0, a very large number of jobs might be dispatched to a host all at once […]  It is not recommended to set this parameter to 0.

Wait a minute. It is not recommended to set this parameter to 0, and the default for LSF 9.1.1 is 0?

If you check what actually happens inside the grid, you will find that yes, jobs start faster, but this has a price. LSF doesn’t have time to check how server load is impacted by these new jobs. LSF simply dispatches all jobs to the SAME server, until the node is full. Only then it sends jobs to another server. The following screenshot shows how all jobs end up running on the same server until it is full (same colors).

New default parameter values for Platform Suite for SAS4

Imagine if the jobs are grid-launched workspace servers. Many sessions will land on the same host all at once. The same users that were happy because their sessions start immediately, soon will complain because they will be contending with each other for resources from the same server.

How can I change this?

To solve this issue, bring back the value of the JOB_ACCEPT_INTERVAL parameter to 1. As with LSF 8, you may want to also lower a bit MBD_SLEEP_TIME and SBD_SLEEP_TIME, but that depends on the actual load on each grid environment. This screenshot shows the same environment running the same jobs, after being tuned. Jobs take a bit more to start, but they are now distributed evenly across all machines.

New default parameter values for Platform Suite for SAS5

The JOB_ACCEPT_INTERVAL parameter can also be set at the queue level, so a more advanced tuning could be to use different values (0, 1 or even more) based on the desired behavior of each queue. This second option is an advanced tuning that usually cannot be implemented during an initial configuration, as it requires careful design, testing and validation.

Changes in SAS 9.4M3

Did you think this long blog was over? Wait, there is more. With SAS 9.4M3 things changed again. This release bundles Platform Suite for SAS version 9.1, which includes LSF 9.1.3. It’s a small change in the absolute number, but the release includes a new parameter:

LSB_HJOB_PER_SESSION Specifies the maximum number of jobs that can be dispatched in each scheduling cycle to each host. LSB_HJOB_PER_SESSION is activated only if the JOB_ACCEPT_INTERVAL parameter is set to 0.

Now you can configure JOB_ACCEPT_INTERVAL=0 to achieve increased grid responsiveness and job throughput, and at the same time put a limit on how many jobs are sent to the same server before LSF starts dispatching them to a different node.

Or you can simply accept the default values for all the parameters: they have changed again, but this time they all fit our recommended practice, as shown in this final table:

New default parameter values for Platform Suite for SAS6

 

 

 

tags: SAS Administrators, SAS Grid Manager, SAS Professional Services

Is your new grid behaving oddly? was published on SAS Users.

4月 302016
 

In a previous blogVisual Analytics audit data collection, I discussed SAS date and time values, and how date and time formats can be used to your advantage in SAS Visual Analytics. That blog addressed some of the features provided for handling date information, like date formats, date hierarchies, and calculated data items based on time intervals.  In this blog, I’ll continue this discussion and share some tips and techniques for working with dates.

Problem: How do I calculate passage of time?

Example: Out of concern over the time taken for delivery of a certain product, we’d like to take a look at the time, from order to delivery of the product, for each order across several years.  The table in memory has data items Product ID, Order ID, Date Ordered, Date Delivered.

Date in Visual Analytics Designer01

Create a new calculated data item, Days from Order to Delivery, calculated as shown:

Date in Visual Analytics Designer02

(1 is added because  we count the order date as a day as well)

Date in Visual Analytics Designer03

Problem: Why does my year have a decimal?

Example: It’s often nice to have a year data item, in addition to the entire date.    The Year operator extracts the year from the date value.|

Date in Visual Analytics Designer04

But then the year displays as shown below, with the comma and decimal! That’s because the new calculated value is a measure and has a default format of COMMA with a width of 12 and 2 decimal places.

Date in Visual Analytics Designer05

To fix this, simply change the format for the year of date data item to Numeric with a width of 12(or less) and no decimals.  Now the year of date values display as below.

Date in Visual Analytics Designer06

And remember, as long as you have a date data item with an associated format that displays year, you don’t need an additional year data item in order to create a hierarchy. Visual Analytics Designer can create the appropriate hierarchy automatically that includes the year.

And if you want to be able to display just the year value, you can duplicate your date data item, and change the format to Year.

Date in Visual Analytics Designer07

Problem: Ugh!  Someone stored these dates as character values!

If dates are stored as character values, that means that when loaded to memory, the date column becomes a category data item with a $ format. With this treatment, you’re unable to take advantage of any of the nice data formats or the date hierarchy features. Plus, when the values are sorted as below, in ascending or descending order, they’re sorted according to character sorting rules, rather than dates.

Date in Visual Analytics Designer08

It’s easy to create a calculated value (we’ll call it RealDate) using the Parse operator, which interprets a string according to a specified informat.

Date in Visual Analytics Designer09

When you click on the format field, you get this selection, and the DATE format with a column width of 9 fits the need here.

Date in Visual Analytics Designer10

Now RealDate displays and sorts as a date should, and since it is now a calculated date value, you can use the different date formats, create date hierarchies, or use the data item with periodic operators.

Date in Visual Analytics Designer11

Problem: The opposite problem!  My date is stored as a date value, so internally, it’s a number, but I want to display the date value as part of a text string!

Text expanded date info below, is a calculated data item, whose expression uses the Concatenation and Format operators.

Date in Visual Analytics Designer12

Both the visual and text forms are shown below:

Date in Visual Analytics Designer13

Problem: How do I create a new calculated date or datetime data item whose values are a certain number of days from the value of another date or datetime data item.

Remember that a date value is stored internally as the number of days since Jan 1, 1960, and a datetime value is the number of seconds since Jan 1, 1960. So this is just a problem of showing passage of time in either days or seconds.

To create a calculated date data item that is X number of days later than the value of date1, create a formula as below, this example calculating a date 21 days later.

Date in Visual Analytics Designer14

To create a calculated datetime date item that is X number of days later than the value of date1, create a formula as below, this example again calculating a date 21 days later. Note that we are multiplying 21 times the number of seconds in a day (60*60*24).

Date in Visual Analytics Designer15

Problem: My date is stored as a number, but the numbers look like this!  YYYYMM (201603, for example)

There are operators available for creating a calculated value that will enable you to overcome most problems of incoming dates being stored in strange ways.

Here is a nice example that was posted in the Global Visual Analytics list for solving this particular problem by using the Mod and Floor operators. Since there was no day information stored with the original numeric date, the day was arbitrarily chosen to be Day 1 of the month:

Date in Visual Analytics Designer16

That's probably enough about ‘dealing with dates’ for one blog. Hopefully, this information has provided some ideas for solving some of the date problems/issues that you might encounter in your work.

tags: SAS Professional Services, SAS Programmers, SAS Visual Analytics

Dealing with Dates in SAS Visual Analytics Designer was published on SAS Users.

4月 282016
 

Look at the report below. Imagine being asked to allow your users to select which Measure, highlighted in yellow, they are looking at: Income, Expense or Profit. This is a frequent report requirement and I’m going to outline just one of the ways you can design your report to satisfy this request using parameters.

In a previous blog, I talked about Using parameters in SAS Visual Analytics reports to prompt users to drive either an aggregated measure or calculated item. This example will allow your users to dynamically select which measures they want to see in their visualizations.

parameters to pick your metric in Visual Analytics Reports1

Steps

  1. Create the custom category to drive the button bar
  2. Create the parameter to hold the button bar’s selection
  3. Create the calculated data item that will hold the selected measure’s value
  4. Add report objects and assign roles

Step 1: Create the custom category to drive the button bar

On the Data tab, use the down menu and select New Custom Category….

Next, from the New Custom Category dialogue, select any category that has a cardinality greater than the number of metric choices you want to give your user. In this example, we will allow our report users to select between Revenue, Expense and Profit, so we have 3 options. Therefore, select a category with a cardinality greater than 3, so 4 or more will work.

parameters to pick your metric in Visual Analytics Reports2

The next part might seem awkward, but this prevents us from having to load a separate table into LASR. If you have the role and capabilities to load data into LASR then you could load a 3-row table that contains one column with the entries Revenue, Expense and Profit, and you could create your control object from that data source. However, you then create the dependency that both tables must be loaded to LASR for the report to work. The technique outlined in this blog allows us to use just this one data source, but restricts the placement of control object. Using this technique means that you cannot use this control object as a report or section prompt. IF you need to use this control as a report or section prompt then you will need to load a separate table.

Otherwise, follow along.

Name your custom category List of Measures. Next, create the labels of the measures you wish your report users to select from. Again, in my example, I want to allow my report users to pick either Revenue, Expense or Profit. Hint: Double click on Label 1 to rename it then use the New label button to add others.

Next, drag and drop at least one value into each custom category label grouping. The custom category data item will only allow you to save it once each category label grouping has a value. You may leave the radio button default Group remaining values as Other selected.

Finally, click OK to save your new custom category.

parameters to pick your metric in Visual Analytics Reports3

Step 2: Create the parameter to hold the button bar’s selection

Now that we have the custom category to feed the button bar’s values, we need to create a parameter to hold the button bar’s selection. From the Data tab, use the drop down menu and select New Parameter….

Select the Type Character and give your parameter a meaningful name. You can leave the current value, or default value, blank. When we assign the parameter to the button bar the value will be populated upon selection.

parameters to pick your metric in Visual Analytics Reports4

Step 3: Create the calculated data item that will hold the selected measure’s value

Here we need to create a calculated data item that will hold the value of the selected measure. This is the measure we will use in our table and graph objects. In pseudo code, we want to create a new measure that

If user selects "Expense" Return <Expense>
Else, If the user selects "Profit" Return <Profit>
Else Return <Revenue>

From the Data tab, use the drop down menu and select New Calculated Item….

parameters to pick your metric in Visual Analytics Reports5

Then use the Visual or Text editor to create a new calculated item named Measure.
Use nested IF…ELSE statements from the Boolean operators and the x = y statement from the Comparison operators.

parameters to pick your metric in Visual Analytics Reports6

Step 4: Add report objects and assign roles

Now we can add our report objects to our report. IMPORTANT: With our technique of using an unrelated category as the source of our custom category we cannot put our button bar in the report or section prompt areas. If we put the button bar in either the report or section prompt areas it would automatically filter the data and we would not get the intended results. DO NOT put the button bar in the report or section prompt area.

In this sample report, you can see I’ve added the Button Bar Control object, the Bar Chart, Crosstab and Line Chart objects to the report area. I want to narrow this section by year, so I have included a Drop-Down List Control to the section prompt area.

parameters to pick your metric in Visual Analytics Reports7

Now let’s look at the role assignments for our various objects. Here is what the report looks like before any style enhancements:

parameters to pick your metric in Visual Analytics Reports8

Drop-Down List Control Object

parameters to pick your metric in Visual Analytics Reports9

Button Bar Control Object

This is where we want to assign our newly created custom category and save that selection to our parameter. This will give us all the categories of our custom category, including the “Other” category.

parameters to pick your metric in Visual Analytics Reports10

Next, we will need to add a filter on this object to remove the “Other” category. From the Filters tab, add a filter for the List of Measures category and deselect the “Other” category.

parameters to pick your metric in Visual Analytics Reports11

Bar Chart, Crosstab and Line Chart

Use our new calculated item as the measure for each of the Bar Chart, Crosstab and Line Chart.

parameters to pick your metric in Visual Analytics Reports12

There is one interaction, where the Bar Chart filters the Crosstab.

parameters to pick your metric in Visual Analytics Reports13

And here is what the report looks like after I’ve altered some of the objects’ styles from the Style tab. I colored the selected bar value to coordinate with the rest of the report objects.
parameters to pick your metric in Visual Analytics Reports14

In this screenshot, I’ve selected the Promotional Product Line.

parameters to pick your metric in Visual Analytics Reports15

Other Applications

As you can envision, this technique can be used for more than just metrics, you could also use this to allow your users to pick a category to use in your objects! Or both.

Just bear in mind that you are restricted to using the Control Objects that support parameters. This includes:

  • Text Input
  • Button Bar
  • Drop-Down List

Slider (single-point only)

 

 

 

 

 

tags: SAS Professional Services, SAS Visual Analytics

Use parameters to pick your metric in Visual Analytics Reports was published on SAS Users.

4月 262016
 

SAS Global ForumWhen was the last time an informational graph or chart caught your eye? I mean, really caught your eye in a way that made you want to emblazon it on a greeting card or frame it for your office?

What’s that…never?

Me neither, until I had the opportunity to see some of the striking visuals and graphics by David McCandless and hear about the thought and passion that goes into his work as a data journalist. McCandless, the author of Knowledge is Beautiful, was a keynote speaker at SAS® Global Forum April 21, an event traditionally focused on the more technical and logistical aspects of analyzing data.

I’m not exaggerating when I say that I was moved by the informative digital images displayed across the conference venue jumbo screens the way some might be moved by a famous painting or sculpture. They revealed depth of understanding and presented analytical findings in such unexpected ways through story, shape, color and connection.

They were beautiful, indeed. But McCandless was quick to point out that it’s important that data visualization transcends aesthetic beauty and aids comprehension

That’s important when you’re faced with billions of numbers and facts. “Images allow us to see something important in a sea of data,” he said. “They tell a story.”

McCandless says the story often lies not in the data points themselves but in the gaps and modulations. “When you visualize data this way, you have a different relationship with it,” he said. “To be able to see it, see the data, helps us understand.”

Many in the analytics world have heard the phrase that data is the new oil, the new fuel to power and motivate business. But McCandless offered a twist on the modern day buzz phrase. “I like to think of data as the new soil,” he said. “Get in and get your hands dirty.” What is revealed could take root and flourish in ways you never imagined.

McCandless also encouraged attendees to give themselves the gift of time and spontaneity when digging into data. “Eject a little play and you may get unexpected results,” he said. He shared visuals created around his own areas of interest, images crafted just for fun, ranging from based-on-truth movies to more than 80 thousand horoscopes. Playing with data is a great way to learn techniques, stretch the imagination, and reveal more memorable ways of sharing business data.

The visual graphics you create may not find a spot above the living room couch, but if they hang in the minds of decision-makers and compel those who rely upon your analysis to change the way they see things, I think McCandless would agree: That is a beautiful thing.

View the full keynote presentation (and catch a glimpse of some of those stunning graphics) on the livestream archive.

 

 

 

 

 

 

 

tags: data visualization, SAS Global Forum

Data is the new soil; get your hands dirty was published on SAS Users.