SAS 9.4

10月 192017
 

The goal of this article is to describe the steps needed to configure the bridge that allows SAS Data Integration 4.902, based on SAS 9.4M4, to load data directly into CAS on SAS Viya 3.2.

Of course, SAS 9.4M5 simplifies this process, as a SAS/CONNECT communication will no longer be required, enabled by a direct function within SAS Data Integration to CAS - but for those of you who may not move immediately from SAS 9.4M4 to SAS 9.4M5, this could be helpful.

It is assumed here that SAS/CONNECT has been installed and configured on both environments, SAS 9.4M4 and SAS Viya 3.2.

Validate the connection from SAS 9.4M4 to SAS Viya 3.2

⇒     Check the status of the SAS/CONNECT Spawner on SAS Viya, on the machine where this service is installed.

SAS Viya 3.2 Bridge for SAS Data Integration Studio

⇒     Note the machine and the listening port of the SAS/CONNECT Spawner on SAS Viya.
⇒     Open SAS Studio on SAS 9.4M4 and sign-in.
⇒     Run the following SAS code with your machine details and a valid SAS Viya user account and check the results.

SAS Viya 3.2 Bridge for SAS Data Integration Studio

⇒     If successful, sign-off the SAS/CONNECT session and sign-out from SAS Studio SAS 9.4M4

Setup SAS9.4M4 metadata

⇒     Open SAS Management Console 9.4M4 as sasadm@saspw.
⇒     On the “Server Manager” plugin, add a “New Server…”

  • Server type: “SAS Application Server”
  • Name: “SASViya”
  • Select “Connect Server” as the sub-type of server you want to add to this SAS Application Server
  • Configure the “Connect Server” as shown below (you might have to create a new authentication domain for SAS Viya) and set the values accordingly (server where the SAS/CONNECT Spawner on SAS Viya is listening)

⇒     On the “Server Manager” plugin, add a “New Server…”

  • Server type: “SAS Cloud Analytic Services Server”
  • Name: “CAS Server”
  • Configure the “CAS Server” as shown below and set the values accordingly (CAS controller)

⇒     On the “User Manager” plugin, set a login for the SASViya application server, on a user or group that you will use in SAS Data Integration Studio

⇒     On the “Data Library Manager” plugin, add a “New Library…”

  • Library type: “SAS Cloud Analytic Services Library”
  • Name: CAS_DATA
  • Assign the library to the SASViya server

⇒     Configure the CAS library as shown below and set the values accordingly (the CASLIB must exist in the SAS Viya environment; here CASPATH is the name of an existing CASLIB).

⇒     Specify the server and connection information as shown below:

Build a SAS Data Integration Studio job to load data into CAS

⇒     Open SAS Data Integration Studio 4.902 as a user who holds, directly or not, a login for the ViyaAuth authentication domain.
⇒     Test the CAS_DATA library by “Register(ing) tables…”

  • In SAS Environment Manager on SAS Viya, some tables must have been loaded before into the CASLIB (the one that is pointed on by the CAS library, here CASPATH), so that you can display some tables in the “Register Tables…” wizard.
  • If you see some tables then it looks like you are all set.

⇒     If you want to go further and test the “Cloud Analytic Services Transfer” DI transformation, create the metadata for a target table in CAS with appropriate columns.

⇒     Build a job that loads a source table to this target table using the “Cloud Analytic Services Transfer” (“Access” folder) as shown below:

⇒     The “Cloud Analytic Services Transfer” is basically a “Data Transfer” transformation that fits with CAS; it enables you to provide some CAS options such as COPIES; by default the table is PROMOTED.

⇒     Run it and check if it ran successfully and if the table is available from the SAS Viya world.

SAS/CONNECT on SAS Viya configuration directory

⇒     The SAS/CONNECT on SAS Viya configuration is located by default here /opt/sas/viya/config/etc/connectserver/default

⇒     You might want to customize/clean some of the files here.

  • Check the CASHOST option in autoexec files, sometimes the value is not appropriate.

Normally, options here are sourced from the CONNECTSERVER_CONFIGURATION in vars.yml during the deployment of SAS Viya.

SAS Viya 3.2 Bridge for SAS Data Integration Studio 4.902 (on SAS 9.4M4) configuration steps was published on SAS Users.

10月 102017
 

Remember when 100MB was large?

SAS 9.4 Maintenance 5 includes new support for reading and writing GZIP files directly. GZIP files, usually found with a .gz file extension, are a different format than ZIP files. Although both are forms of compressed files, a GZIP file is usually a compressed copy of a single file, whereas a ZIP file is an "archive" -- a collection of files in a compressed virtual folder. GZIP tools are built into Unix/Linux platforms and are commonly used to save space when storing large text-based files that you're not ready to part with: log files, csv files, and more. The algorithm used to compress GZIP files performs especially well with text files, although you can technically GZIP any file that you want.

I've written extensively about using FILENAME ZIP to read and write ZIP archives with SAS. The latest version of filename my_gz ZIP "path-to-file/compressedfile.txt.gz" GZIP;

Here's an example that creates a compressed version of a log file:

filename source "C:\Logs\SEGuide_log.10168.txt";
filename tozip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
 
data _null_;   
   infile source;
   file tozip ;
   input;
   put _infile_ ;
run;

In my test here, the result represents a significant size difference, with the compressed file occupying just 14% of the space.


To "re-inflate" the compressed file, we can perform the opposite operation. (I added the ENCODING option here because I know my log file was UTF-8 encoded.)

filename target "C:\LogsExpanded\SEGuide_log.10168.txt" encoding='utf-8';
filename fromzip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
 
data _null_;   
   infile fromzip;
   file target ;
   input;
   put _infile_ ;
run;

You don't have to explicitly expand a compressed text file in order to read it with SAS. You can use the GZIP method to read and parse a .gz file directly, similar to the zcat command that you might be familiar with from the Unix shell:

filename fromzip ZIP "C:\Logs\SEGuide_log.10168.txt.gz" GZIP;
data logdata;   
   infile fromzip; /* read directly from compressed file */
   input  date : yymmdd10. time : anydttme. ;
   format date date9. time timeampm.;
run;

If your file is in a binary format such as a SAS data set (sas7bdat) or Excel (XLS or XLSX), you probably will need to expand the file completely before reading it as data. These files are read using special drivers that don't process the bytes sequentially, so you need the entire file available on disk.

Note: Because each GZIP file represents just one compressed file, the MEMBER= option doesn't apply. When dealing with ZIP file archives that contain multiple files, you could use the MEMBER= option on FILENAME ZIP to address a specific file that you want. My recent example about FINFO and file details relies heavily on that approach. However, the GZIP option and MEMBER= options are mutually exclusive. In that way, it's much simpler...just like its Unix shell equivalent.


* ZIP drive image By © Raimond Spekking / CC BY-SA 4.0 (via Wikimedia Commons), CC BY-SA 4.0, Link

The post Reading and writing GZIP files with SAS appeared first on The SAS Dummy.

8月 182017
 

If you are a SAS administrator managing an environment on UNIX or z/OS, you must use the sas.servers script on a regular basis. As you know, one of its uses is to display the current status of all servers. Is the output accurate? Absolutely. Is it easy to read? Relatively. Is it visually attractive? Not so much.

Human beings are visual creatures. Conventional wisdom says that a picture speaks a thousand words. However, I am not using images to improve the output, but another powerful tool: color. According to this Xerox paper, color captures attention and enhances productivity. It can improve search time, reduce errors, and increase comprehension. As a result, this blog post provides the steps for applying color and an easy-to-read layout to make the sas.servers script look cute and even fun!

Preliminary Steps

Before we begin, it is important to give you some recommendations:

  1. Stop all SAS services.
  2. Backup the files: sas.servers, sas.servers.mid and sas.servers.pre
  3. Apply these changes to a Development or Testing environment.

As a matter of fact, I consider steps 1 and 3 as optional. This script is only used by the SAS platform to start/stop/restart or check the status of the servers. Moreover, if you follow all steps carefully, you can apply these changes safely in a Production environment. In contrast, step 2 is relevant; it is always a good practice to backup essential files before modifying them. In case you need to rollback, you can restore them easily and quickly.

Things to Consider

The About the sas.servers Script section from the SAS 9.4 Intelligence Platform: System Administration Guide provides a caution message: "You should not directly update the sas.servers script." In our case, the type of customization we are about to perform requires a manual update. Don't worry, your script is in good hands.

Furthermore, if you ever need to update the sas.servers script because you want to add/remove a server, you have to run the generate_boot_scripts.sh script to regenerate this file. After doing so, you can lose all the changes made in this post. Keep this in mind, so you can backup the current files before attempting this task.

Easy-to-read Layout

Let's jump into the details. First of all, let's define the new layout of the desired output. Grab your favorite text editor. I use Notepad++ at the office, and Sublime Text at home. Go to your /SASCONFIG/Lev1/ directory and run a regular status command as the sas user. Assuming you followed the preliminary steps and depending on the products installed and the number of SASServers deployed, you should get a similar output:

./sas.servers status
SAS servers status:
SAS Web Infrastructure Data Server is NOT up
SAS Metadata Server 1 is NOT up
SAS Object Spawner 1 is NOT up
SAS DIP Job Runner 1 is NOT up
SAS Information Retrieval Studio Server is NOT up
SAS JMS Broker is NOT up
SAS Cache Locator Service ins_41415 is NOT up
SAS Web Server is NOT up
SAS Web App Server SASServer1_1 is NOT up
SAS Environment Manager is NOT up
SAS Environment Manager Agent is NOT up

Copy that output, except the first line, and paste it into your text editor. Here you can modify the output according to your taste. What I did was to identify the longest line, which is SAS Information Retrieval Studio Server, then I added four spaces to the right and substituted the legend is NOT up for [DOWN]. Likewise, I applied these changes to the rest of the servers, removed the instance number, and kept them all aligned:

SAS Web Infrastructure Data Server         [DOWN]
SAS Metadata Server                        [DOWN]
SAS Object Spawner                         [DOWN]
SAS DIP Job Runner                         [DOWN]
SAS Information Retrieval Studio Server    [DOWN]
SAS JMS Broker                             [DOWN]
SAS Cache Locator Service ins_41415        [DOWN]
SAS Web Server                             [DOWN]
SAS Web App Server SASServer1_1            [DOWN]
SAS Environment Manager                    [DOWN]
SAS Environment Manager Agent              [DOWN]

I chose the pair [ UP ]/[DOWN] to reflect the status because I wanted the look and feel from CentOS 6 boot process. If you are/were a CentOS 6 user, you can remember that the services display the legends: [ OK ] or [FAILED] when booting. You are free to use other alternatives such as ACTIVE/INACTIVE or perhaps RUNNING/STOPPED with or without brackets. Now that the layout is finished, let's move on to the color department.

All You Need Is Color

The fun finally arrived. Let's integrate the layout into our three scripts and add the main ingredient: color!

Again, I am assuming at this stage that you already backed up the files: sas.servers, sas.servers.mid, and sas.servers.pre. Next, open the sas.servers file with your vi editor and add this code anywhere at the top, below the block of comments:

#*****
# Custom Colors for status command
# RED   for [DOWN]
# GREEN for [ UP ]
# NC    for No Color
#*****
RED='\e[31m'
GREEN='\e[32m'
NC='\e[0m'

These lines create three variables with three different color codes: a RED variable with code 31, a GREEN variable with code 32, and a NC variable with the default color code. The definition of these variables is optional but recommended, since they help you debug problems or change colors more easily. If you prefer to use different colors or attributes, you can play with the codes as shown in Bash tips: Colors and formatting.

Considering that the sas.servers script does not contain all servers, you have to add the same code to sas.servers.mid and sas.servers.pre files.

First Example

Now that we have defined the three variables in all the necessary files, let's use them. I'll show you how to apply them to a couple of servers, and then you can replicate it to the rest. The first one is the SAS Metadata Server. Open the sas.servers script again and find these lines:

# SAS Metadata Server
SASMETA_WONT_START_OTHERS="The remaining SAS servers will NOT be started as a result."
 
SASMETA1_IS_UP="SAS Metadata Server 1 is UP"
SASMETA2_IS_UP="SAS Metadata Server 2 is UP"
SASMETA3_IS_UP="SAS Metadata Server 3 is UP"
SASMETA4_IS_UP="SAS Metadata Server 4 is UP"
SASMETA5_IS_UP="SAS Metadata Server 5 is UP"
 
SASMETA1_IS_DOWN="SAS Metadata Server 1 is NOT up"
SASMETA2_IS_DOWN="SAS Metadata Server 2 is NOT up"
SASMETA3_IS_DOWN="SAS Metadata Server 3 is NOT up"
SASMETA4_IS_DOWN="SAS Metadata Server 4 is NOT up"
SASMETA5_IS_DOWN="SAS Metadata Server 5 is NOT up"

Since I have a single Metadata Server instance, the only meaningful variables are SASMETA1_IS_UP and SASMETA1_IS_DOWN. Delete their values, copy the correct string from your text editor, and paste it in both variables. Fix them accordingly:

SASMETA1_IS_UP="SAS Metadata Server                        [ UP ]"
#more instances
 
SASMETA1_IS_DOWN="SAS Metadata Server                        [DOWN]"
#more instances

The final touch is to give color to our script. Use the GREEN and RED variables for the UP/DOWN statuses. It is important to include the NC variable at the end to remove all attributes:

SASMETA1_IS_UP="SAS Metadata Server                        [ ${GREEN}UP${NC} ]"
#more instances
 
SASMETA1_IS_DOWN="SAS Metadata Server                        [${RED}DOWN${NC}]"
#more instances

Second Example

The process is the same for all the servers defined in the sas.servers script. For the other two scripts it is a little different, but still quite easy. I am going to use the SAS Web Infrastructure Data Server as the second example. Open the sas.servers.pre script and look for the server_status() function. Pay attention to these lines:

       if [ $? -eq 0 ]; then
          # Server is already running
          echo "SAS Web Infrastructure Data Server is UP"
       else
          echo "SAS Web Infrastructure Data Server is NOT up"
       fi
    }
    else
      echo "SAS Web Infrastructure Data Server is NOT up"

A subtle difference from the previous example is the echo command. In the sas.servers script, there is a logmsg() function that uses the "echo -e" command behind the scenes. In this case, we have to explicitly add the -e option to enable the interpretation of backslash escapes. Let's also integrate the color and layout:

          echo -e "SAS Web Infrastructure Data Server         [ ${GREEN}UP${NC} ]"
       else
          echo -e "SAS Web Infrastructure Data Server         [${RED}DOWN${NC}]"
       fi
    }
    else
      echo -e "SAS Web Infrastructure Data Server         [${RED}DOWN${NC}]"

At this point, with the above examples, you should have a solid idea about the required changes to accomplish our goal. Now it is your turn to apply them to the rest of the servers.

Finished Product

If you followed this article in detail and performed the steps in all the required servers, your output should resemble mine:

Get the green light by running the start command:

Final Thoughts

I am a visual person with a curious mind. One of the things I like is to customize the tools I use the most, so I decided to make the sas.servers script output a little more attractive to my eyes. I hope you liked the result. If you are still not sure whether to implement this idea or not, let's suppose there are some problems with a couple of servers in your environment and they stop running. Which output would you rather look at? Which one is easier to spot an issue? Let me know your thoughts in the comments below, or even better you can share your creative outputs!

Making the sas.servers script look pretty was published on SAS Users.

2月 102017
 

Since the SAS 9.4 M2 release in December 2014, there have been several refinements and updates to the middle tier that are of interest to installers and administrators. In this blog, I’m going to summarize them for you. What I’m describing here is available in the newest SAS release (9.4 M4). I’ll describe them at a high level, and refer you to the documentation for details and how to implement some of these changes.

Security enhancements

Preserve your TLS Customizations:
For security purposes, many of you will manually add TLS configurations, either to the SAS Web Server, the SAS Web Application Server, or both. In addition, you may prefer to use your own reverse proxy server (such as IIS), either instead of, or in addition to, the SAS Web Server. Before the 9.4 M4 release, when upgrading or applying maintenance, you had to undo these custom configurations, perform the upgrade, and then apply the custom configurations again. Now, the upgrade will preserve them, making the process much easier. See Middle-Tier Security in the Middle Tier Administration Guide, Fourth Edition for full details.

Newer versions of OpenSSL are now provided (see doc for specific version numbers):
A Java upgrade enables enforcement of TLSv2. TLS is now considered the security standard for https connections, (SSL is obsolete) and this can be enforced with configurations to the SAS Web Server and the SAS Web Application Server. The new version of Java SAS is using (Ver 1.7+) now allows for this. One important thing to be aware of is that certificates are completely independent of which protocol you are using, and therefore any certificates you may have been using with SSL should work equally with newer TLS protocols.

Management of the trusted CA (Certificate Authority) bundle:
SAS now has a trusted CA bundle, that can be managed by the SAS Deployment Manager, in a new location:  SASHome/SASSecurityCertificateFramework/1.1/cacerts/. The CA certificates can be root certificates, intermediate certificates, or both. Here’s what the menu item looks like:

Middle Tier Changes and Upgrades in SAS 9.4 M4

Previously it was necessary to manually add your root/intermediate certificates to the Java truststore “cacerts,” located inside the JRE; now it’s done through the new interface. If you are on Windows, you must also add trusted CAs to the Windows store (as before), which will make them available to any browsers running there. This is documented at http://www.sqlservermart.com/HowTo/Windows_Import_Certificate.aspx and elsewhere online.

Security Support for SAS Web Applications – white list external sites, and HTTP request methods:
For added security, web sites hosting SAS web applications can now maintain a white list of external URLs that are allowed to connect in. This provides protection against Cross Site Request Forgeries, and other vulnerabilities. This is what the prompt looks like in the SDW:

Middle Tier Changes and Upgrades in SAS 9.4 M4

HTTP request methods can also be specified as allowed/not allowed. The list of URLs can be specified during installation in the SDW (shown above), or using the SAS Management Console. You can disable whitelist checking entirely, and you can add a “blacklist” or specific sites to always block. You can also block based on request method–ie, GET, POST, PUT, etc. See the Middle Tier Administration Guide for details.

Forward Proxy Configuration:
You can now set up SAS web applications to forward external URL requests through a proxy–here it’s called a forward proxy server. Many organizations do this behind their firewalls. See details for how to set this up in the administration guide.

Other miscellaneous changes:
As an administrator you can now force users to Log Off using SAS Web Administration Console.    You can also send emails to one or more users from the same window.  This is what the menu looks like:

Middle Tier Changes and Upgrades in SAS 9.4 M4

Faster start-up time for the SAS Web Application Server

JMS Broker (ActiveMQ) now uses Version 5.12.2 (fixed bugs).

SAS Web Server now uses version 5.5.2 and includes an updated mod_proxy_connect module for TLS tunneling.

References

SAS 9.4 Intelligence Platform: Middle Tier Administration Guide, Fourth Edition

Encryption in SAS 9.4, Sixth Edition

 

tags: SAS 9.4, SAS Administrators, SAS Professional Services, security

Middle Tier Changes and Upgrades in SAS 9.4 M4 was published on SAS Users.

1月 202017
 

SAS/GRAPH 9.4 capabilitiesI remember my grandparents talking about how hard things were for them growing up. They would say, “Things were so bad that we had to walk uphill, both ways, in the freezing snow to get to school.” It was always hard for me to relate to these statements because the school bus picked me up at the end of my driveway. Fast forward to today and people are riding on hoverboards. Through the years, advancement in transportation has made it easier for us to get where we need to be.

SAS/GRAPH® Version 6

The evolution of SAS/GRAPH® is similar. In the earlier days of SAS® software, during Version 5 and 6 of SAS/GRAPH, I understood how difficult it was to create some of the graphs that customers wanted. A customer recently asked whether I could send him the code that produced the graph below, which he found in the SAS/GRAPH® Software: Reference, Volume 1 Version 6 Edition:

sasgraph-9-4-capabilities

In Version 6, the only way to create this graph, referred to as a butterfly chart, was by using SAS/GRAPH and the Annotate facility. The annotation statements added over 60 lines of code to the program.

Below is a snippet of the Version 6 program that created the bars on the left side of the graph.

Click this link to see the entire Version 6 program.

     /* female bars on left */
    %bar(39.8, 10.5, 25.0, 20.0, blue, 0, solid);
    %bar(39.8, 20.71, 15.0, 30.7, green, 0, solid);
    %bar(39.8, 31.42, 10.0, 41.42, red, 0, solid);
    %bar(39.8, 42.14, 32.0, 52.14, blue, 0, solid);
    %bar(39.8, 52.85, 33.0, 62.85, green, 0, solid);
    %bar(39.8, 63.57, 36.0, 73.57, red, 0, solid);
    %bar(39.8, 74.28, 35.0, 84.28, blue, 0, solid);
    %bar(39.8, 85.0, 33.0, 95.0, green, 0, solid);

SAS/GRAPH® 9.4

Fast forward to SAS® 9.4. ODS Graphics and SG procedures have been part of Base SAS® since SAS® 9.3, making it much easier to create high-quality graphs without using additional software. The graph that was previously created using the Annotate facility can now be created using the Graph Template Language (GTL) and the SGRENDER procedure. Here is the program to create the entire graph:

Note: The program below contains numbered annotations that correspond to a discussion below the program. So, if you copy and paste this program into SAS, be sure to delete the annotations before running this code.

data ratio;
  input Age $1-8 type $ Female Male;
datalines;
over 79  A 19 18
70-79    B 15 14
60-69    C 13 12
50-59    A 24 46
40-49    B 26 18
30-39    C 92 61
20-29    A 77 88
under 20 B 42 100
;
run;
 
proc template; 
   define statgraph population; ❶
   begingraph /border=false❷ datacolors=(green blue red); ❸
        entrytitle 'Population Tree' / textattrs=(size=15);❹
   entrytitle 'Distribution of Population by Sex';
   layout lattice ❺/ columns=2 ❻ columnweights=(.55 .45); ❼
      layout overlay ❽/ walldisplay=none y2axisopts=(reverse=true
                         tickvaluehalign=center 
                         display=(tickvalues))  
                         xaxisopts=(displaysecondary=(label) 
                         display=(tickvalues line) reverse=true 
                         labelattrs=(weight=bold));
         barchart category=age response=Female / group=type orient=horizontal 
                                                 yaxis=y2 barlabel=true;
      endlayout;
      layout overlay ❽/ walldisplay=none 
                         yaxisopts=(reverse=true display=none)
                         xaxisopts=(displaysecondary=(label)   
                         display=(tickvalues line)
                         labelattrs=(weight=bold)); 
         barchart category=age response=Male / group=type orient=horizontal 
                                               barlabel=true ;
      endlayout;
   endlayout;
   endgraph;
   end;
 
proc sgrender data=ratio template=population;
run;

As you can see, this program is much simpler than the one created using Version 6 above. Let’s take a closer look at the code.

❶ The TEMPLATE procedure creates a GTL definition called POPULATION with the DEFINE statement.

❷ In the BEGINGRAPH statement, the BORDER=FALSE option turns off the outside border.

❸ The DATACOLORS option defines the colors for the groups.

❹ The ENTRYTITLE statements define the titles for the graph.

❺ A LAYOUT LATTICE block serves as a wrapper for the LAYOUT OVERLAY statements.

❻ The COLUMNS option in the LAYOUT LATTICE block defines the layout of the cells.

❼ Because the graph in the left cell contains the bar values, the COLUMNWEIGHTS option allocates more room for this graph. The values for COLUMNWEIGHTS need to add up to 1.

❽ Two LAYOUT OVERLAY statements define the two cells in this graph.

LAYOUT OVERLAY Code

The two LAYOUT OVERLAY blocks are similar, so we will look closer at only the first one:

layout overlay / walldisplay=none ❶ 
                 y2axisopts=(reverse=true 
                 tickvaluehalign=center display=(tickvalues)) ❷ 
                 xaxisopts=(displaysecondary=(label)display=(tickvalues line)reverse=true ❺ 
                 labelattrs=(weight=bold));
   barchart category=age response=Female ❻/ group=type ❼
                           orient=horizontal ❽ yaxis=y2 ❾ barlabel=true; ❿
endlayout;

❶ The WALLDISPLAY=NONE option turns off the border around the graph.

❷ The REVERSE=TRUE option reverses the Y axis order, TICKVALUEHALIGN=CENTER centers the tick mark values, and the DISPLAY=TICKVALUES option displays only the tick mark values. These options are specified within Y2AXISOPTS.

❸ DISPLAYSECONDARY=(LABEL) displays the X axis label on the X2axis, at the top of the graph.

❹ On the X axis, at the bottom of the graph, the DISPLAY=(TICKVALUES LINE) option displays the tick mark values and the axis line.

❺ The REVERSE=TRUE option also reverses the X axis.

❻ The bar chart contains a bar for each Age. The length of the bars is based on the values of the variable Female with the CATEGORY=AGE and RESPONSE=FEMALE options in the BARCHART statement respectively.

❼ The group variable, TYPE, determines the color of the bars.

❽ The ORIENT=HORIZONTAL option in the BARCHART statement specifies that the bars are horizontal.

❾ YAXIS=Y2 specifies that the values are plotted against the right Y axis, Y2 axis.

❿ The BARLABEL=TRUE option provides labels for the bars.

Here is the graph that is created when you submit this program:

sasgraph-9-4-capabilities02

As this example shows, SAS graphing capabilities have improved over the years just as transportation options have progressed. Be sure to take advantage of these improvements to create helpful visualizations of your data! If you would like to create a similar graph with PROC SGPLOT, refer to Sanjay Matange’s blog post “Butterfly plots.”


Version 6 program

    /* set the graphics environment */
 goptions reset=global gunit=pct border
          ftext=swissb htitle=6 htext=3 dev=png;
 %annomac;
 
    /* create the Annotate data set, POPTREE */
 data poptree;
       /* length and type specification */
    %dclanno;
 
       /* set length of text variable   */
    length text $ 16;
 
 
       /* window percentage for x and y */
    %system(5, 5, 3);
 
       /* draw female axis lines */
    %move(5, 10);
    %draw(40, 10, red, 1, .5);
    %draw(40, 95, red, 1, .5);
 
       /* draw male axis lines */
    %move(56.1, 95);
    %draw(56.1, 10, red, 1, .5);
    %draw(95, 10, red, 1, .5);
 
       /* label categories */
    %label(75.0, 97.0, 'Male', green, 0, 0, 4, swissb, 5);
 
       /* at top */
    %label(25.0, 97.0, 'Female', green, 0, 0, 4, swissb, 5);
    %label(5.0, 5, '100', blue, 0, 0, 4, swissb, 5);
    %label(22.5, 5, ' 50', blue, 0, 0, 4, swissb, 5);
    %label(40.0, 5, ' 00', blue, 0, 0, 4, swissb, 5);
    %label(95.0, 5, '100', blue, 0, 0, 4, swissb, 5);
    %label(75.0, 5, ' 50', blue, 0, 0, 4, swissb, 5);
    %label(56.0, 5, ' 00', blue, 0, 0, 4, swissb, 5);
 
       /* label age */
    %label(48.0, 15.25, 'under 20', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 25.0, '20 - 29', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 36.7, '30 - 39', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 47.4, '40 - 49', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 57.8, '50 - 59', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 68.6, '60 - 69', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 79.3, '70 - 79', blue, 0, 0, 4, swissb, 5);
    %label(48.0, 90.0, 'over 79', blue, 0, 0, 4, swissb, 5);
 
       /* male bars on right */
    %bar(56.2, 10.5, 95.0, 20.0, blue, 0, solid);
    %bar(56.2, 20.71, 90.0, 30.71, green, 0, solid);
    %bar(56.2, 31.42, 80.0, 41.52, red, 0, solid);
    %bar(56.2, 42.14, 62.0, 52.14, blue, 0, solid);
    %bar(56.2, 52.85, 72.0, 62.85, green, 0, solid);
    %bar(56.2, 63.57, 60.0, 73.57, red, 0, solid);
    %bar(56.2, 74.28, 61.0, 84.28, blue, 0, solid);
    %bar(56.2, 85.0, 63.0, 95.0, green, 0, solid);
 
       /* label male bars on right */
    %label(95.0, 20.0, '100', black, 0, 0, 4, swissb, 7);
    %label(90.0, 30.71, '88', black, 0, 0, 4, swissb, 7);
    %label(80.0, 41.52, '61', black, 0, 0, 4, swissb, 7);
    %label(62.0, 52.14, '18', black, 0, 0, 4, swissb, 7);
    %label(72.0, 62.85, '46', black, 0, 0, 4, swissb, 7);
    %label(60.0, 73.57, '12', black, 0, 0, 4, swissb, 7);
    %label(61.0, 84.28, '14', black, 0, 0, 4, swissb, 7);
    %label(62.0, 95.0, '18', black, 0, 0, 4, swissb, 7);
 
       /* female bars on left */
    %bar(39.8, 10.5, 25.0, 20.0, blue, 0, solid);
    %bar(39.8, 20.71, 15.0, 30.7, green, 0, solid);
    %bar(39.8, 31.42, 10.0, 41.42, red, 0, solid);
    %bar(39.8, 42.14, 32.0, 52.14, blue, 0, solid);
    %bar(39.8, 52.85, 33.0, 62.85, green, 0, solid);
    %bar(39.8, 63.57, 36.0, 73.57, red, 0, solid);
    %bar(39.8, 74.28, 35.0, 84.28, blue, 0, solid);
    %bar(39.8, 85.0, 33.0, 95.0, green, 0, solid);
 
       /* label female bars on left */
    %label(25.0, 20.0, '42', black, 0, 0, 4, swissb, 9);
    %label(15.0, 30.7, '77', black, 0, 0, 4, swissb, 9);
    %label(10.0, 41.42, '92', black, 0, 0, 4, swissb, 9);
    %label(32.0, 52.14, '26', black, 0, 0, 4, swissb, 9);
    %label(33.0, 62.85, '24', black, 0, 0, 4, swissb, 9);
    %label(36.0, 73.57, '13', black, 0, 0, 4, swissb, 9);
    %label(35.0, 84.28, '15', black, 0, 0, 4, swissb, 9);
    %label(33.0, 95.0, '19', black, 0, 0, 4, swissb, 9);
 run;
 
    /* define the titles */
 title1 'Population Tree';
 title2 h=4 'Distribution of Population by Sex';
 
    /* generate annotated slide */
 proc gslide annotate=poptree;
 run;
 quit;
tags: Problem Solvers, SAS 9.4, SAS Programmers

Comparing SAS/GRAPH® 9.4 capabilities with SAS/GRAPH® Version 6 was published on SAS Users.

1月 122017
 

analytics resolutionsThe holiday season is over – and you survived. You’ve made a lot of personal resolutions for 2017 - go to the gym, eat less sugar, save more money, visit Grandma more often. These are all great personal resolutions for 2017, but what about your analytics resolutions? If you are having trouble with your analytics resolutions then let us help you out. The recent release of SAS 9.4 M4 will help you make 2017 your best analytics year yet.

Resolution 1: Build more accurate models faster!

Now you will be able to leverage the power of the two most advanced analytics platforms on the market, SAS 9 and SAS Viya from one interface. Using SAS/Connect, users can call powerful SAS Viya analytics from within a process flow in Enterprise Miner. Would you prefer to use the super-fast, autotuned gradient boosting in SAS Viya? No problem! Call SAS Viya analytics directly from Enterprise Miner using the SAS Viya Code node. Then, from the same process flow you can also call open source models, all from one interface, SAS Enterprise Miner. Do you prefer to use SAS Studio on SAS 9? You will also be able to call SAS Viya analytics from SAS Studio as well. With SAS 9 M4, SAS gives you the ability to use both of SAS’ powerful platforms from one interface.

Resolution 2: Score your unstructured models in Hadoop without moving your data!

Got Hadoop? Got a lot of unstructured data? Now SAS Contextual Analysis allows you to score models in Hadoop using the SAS Code Accelerator add-on. Identify new insights with your unstructured text without ever having to move your data. Score it all in Hadoop. Uncover new trends and topics buried in documents, emails, social media and other unstructured text that is stored in Hadoop. You will be able to do it faster because you won’t have to move that data outside of Hadoop. SAS just keeps getting better in 2017.

Resolution 3: Make better forecasts using the weather!

Through SAS/ETS, econometricians and others wanting to incorporate weather data into their models can now do so directly through two new interface engines. SASERAIN enables SAS users to retrieve weather data from the World Weather Online website. And SASENOAA provides access to severe weather data from the National Oceanic and Atmospheric Administration (NOAA) Severe Weather Data Inventory (SWDI) web service. So now you’ll know why there was that big sales spike for rock salt and snow shovels in July! Who says there is no climate change in 2017?

Resolution 4: Estimate causal effects more efficiently!

The new CAUSALTRT procedure in SAS/STAT estimates the average causal effect of a binary treatment variable T on a continuous or discrete outcome Y. Depending on the application, the variable T can represent an intervention (such as smoking cessation – which is a great 2017 resolution - versus control), an exposure to a condition (such as attending private versus public schools), or an existing characteristic of subjects (such as high versus low socioeconomic status). The CAUSALTRT procedure estimates two types of causal effects: the average treatment effect and the average treatment effect for the treated. And best of all, the causal inference methods that the CAUSALTRT procedure implements are designed primarily for use with data from nonrandomized trials or observational studies, where you observe T and Y without assigning subjects randomly to the treatment conditions.

Resolution 5: Design better factory floors!

A factory floor can be a complicated place, with raw materials coming in one side, and finished products going out the other. Options are virtually unlimited for the placement of materials and equipment – and a poorly designed layout can dramatically reduce production capability. Yet experimenting with different layouts would be extremely costly and time consuming. Thankfully, SAS Simulation Studio (a component of SAS/OR) provides a rich – and animated – environment for testing alternatives and coming up with the most appropriate design. And it can handle any kind of discrete-event simulation, integrating with JMP for experimental design and input analysis, and with JMP and SAS for source data and analysis of simulation results. How will your factory floor simulation impact your productivity in 2017?

tags: analytics, SAS 9.4, SAS Viya

Five great analytics resolutions for 2017 was published on SAS Users.

12月 212016
 

The report-ready SAS Environment Manager Data Mart has been an invaluable addition to SAS 9.4 for SAS administrators. The data mart tables are created and maintained by the SAS Environment Manager Service Architecture Framework and provide a source of data for out-of-the box reports as well as custom reports that any SAS administrator can easily create. As you can imagine, the size of the tables in the data mart can grow quite large over time so balancing the desired time span of reporting and the size of the tables on disk requires some thought. The good news: SAS 9.4 M4 has made that job even easier.

The Environment Manager Data Mart (EVDM) has always provided a configuration setting to determine how many days of resource records to keep in the data mart tables. You can see below that in a fresh SAS 9.4 M4 installation, the default setting for “Number of Days of Resource Records in Data Mart” is set to 60 days. This means that EVDM data records older than 60 days are deleted from tables whenever the data mart ETL process executes.

EV Data Mart Tables in 9.4M4

The space required to house the Environment Manager Data Mart is split across three primary areas.

  • The ACM library tables contain system level information
  • The APM library tables contain audit and performance data culled from SAS logs
  • The KITS library tables contains miscellaneous tables created by data mart kits that collect specialty information about HTTP access, SAS data set access, and such.

Prior to SAS 9.4M4, the ACM and APM libraries duly archived data according to the “Number of Days of Resource Records in Data Mart” setting, but the KITS library did not. For most of the KITS tables this is not such a big deal but for some deployments, the HTTPACCESS table in the KITS library can grow quite large. For administrators who have enabled the VA feed for the Service Architecture Framework, the size of the HTTPACCESS table directly impacts the time it takes to autoload the results of each refresh of the data mart, as well as the amount of memory consumed by the LASR Server used for the Environment Manager Data Mart LASR library.

So what is the big change for SAS 9.4 M4?

The KITS library now respects the “Number of Days of Resource Records in Data Mart” setting and removes data older than the threshold.  If you are a SAS administrator, you can now forget about having to separately manage the KITS library which should simplify space management.

SAS administrators may need to adjust the “Number of Days of Resource Records in Data Mart” setting to strike a balance between the date range requirements for reporting and the amount of disk space they have available for storing the EVDM tables.  With SAS 9.4 M4, however, administrators can rest assured that all EVDM tables will self-manage according to their wishes.

More on the Service Architecture Framework.

tags: SAS 9.4, SAS Administrators, SAS Environment Manager, SAS Professional Services

Easier Space Management for EV Data Mart Tables in 9.4M4 was published on SAS Users.

3月 052016
 

In previous articles, I've shared tips about how you can work with SAS and ZIP files without requiring an external tool like WinZip, gzip, or 7-Zip. I've covered:

But a customer approached me the other day with one scenario I missed: how to add SAS data sets to an existing ZIP file. It's a variation of a tip that I've already shared, but with two differences. First, in order to add a data set to a ZIP file, you have to know its physical filename -- not just the LIBNAME.MEMBER reference that you use in SAS procedure steps. And second, I had not shown how to add a new file to an existing ZIP archive -- though it turns out that's pretty simple.

Find the file name for a SAS data set

There are several ways to do this. For my approach, I used the output from PROC CONTENTS. Notice that I had to capture the ODS output (not the OUT= data set) to grab the file name. I wrapped it in a macro for easy reuse. And since I ultimately need a SAS fileref to map to the path, I've assigned one (data_fn) in my macro.

/* macro to assign a fileref to a SAS data set in a Base library */
%macro assignFilerefToDataset(_dataset_name);
    %local outDsName;
    ods output EngineHost=File;
    proc contents data=&_dataset_name.;
    run;
    proc sql noprint;
        select cValue1 into: outDsName 
            from work.file where Label1="Filename";
    quit;
    filename data_fn "&outDsName.";
%mend;

How to add a new member to a ZIP file

Now that I have the source file, I need to designate a destination file in a ZIP archive. The FILENAME ZIP method will create a new ZIP file if one does not yet exist, or it can add to an existing ZIP. To ensure I'm starting from scratch, I assign a simple fileref to my target destination and then delete the file.

/* Assign the fileref - basic file method */
filename projzip "&projectDir./project.zip";
/* Start with a clean slate - delete ZIP if it exists */
data _null_;
    rc=fdelete('projzip');
run;

To create a new ZIP file and designate a path and file name within it, I used the FILENAME ZIP method with the MEMBER= option. Note that I specified the "data/" subfolder in the MEMBER= value; this will place the file into a named subfolder within the archive.

/* Use FILENAME ZIP to add a new member -- CLASS */
/* Put it in the data subfolder */
filename addfile zip "&projectDir./project.zip" 
    member='data/class.sas7bdat';

Then finally, I need to actually "copy" the file into the archive. I do this by streaming the source file into the target fileref byte-by-byte:

/* byte-by-byte copy */
/* "copies" the new file into the ZIP archive */
data _null_;
    infile data_fn recfm=n;
    file addfile recfm=n;
    input byte $char1. @;
    put  byte $char1. @;
run;
 
filename addfile clear;

That's it! I now have a ZIP file with one member entry. Now I can "press repeat" to add a second entry:

%assignFilerefToDataset(sashelp.cars);
/* Use FILENAME ZIP to add a new member -- CARS */
/* Put it in the data subfolder */
filename addfile zip "&projectDir./project.zip" 
    member='data/cars.sas7bdat';
/* byte-by-byte copy */
/* "copies" the new file into the ZIP archive */
data _null_;
    infile data_fn recfm=n;
    file addfile recfm=n;
    input byte $char1. @;
    put  byte $char1. @;
run;
 
filename addfile clear;

Optional: Report on the ZIP file contents

If I want to report on the total contents of the ZIP file now, here's a DATA step and PROC CONTENTS step that does the job:

/* OPTIONAL for reporting */
/* Report on the contents of the ZIP file */
/* Assign a fileref wth the ZIP method */
filename inzip zip "&projectDir./project.zip";
/* Read the "members" (files) from the ZIP file */
data contents(keep=memname);
    length memname $200;
    fid=dopen("inzip");
    if fid=0 then
        stop;
    memcount=dnum(fid);
    do i=1 to memcount;
        memname=dread(fid,i);
        output;
    end;
    rc=dclose(fid);
run;
/* create a report of the ZIP contents */
title "Files in the ZIP file";
proc print data=contents noobs N;
run;

Result:

Files in the ZIP file 

memname
---------------------
data/class.sas7bdat
data/cars.sas7bdat 
N = 2

I hope that this helps to make the FILENAME ZIP method more useful to those who want to try it out. I'm sure that there will be more scenarios that people will ask about; someday, if I write enough blog posts, I'll have it all covered!

Sample program: You can view/download the entire SAS program (containing the snippets I've featured and more) from my GitHub profile.

tags: FILENAME ZIP, SAS 9.4, SAS programming, ZIP files

The post Add files to a ZIP archive with FILENAME ZIP appeared first on The SAS Dummy.

9月 292015
 

Thanks to the proliferation of cloud services and REST-based APIs, SAS users have been making use of PROC HTTP calls (to query these web services) and some creative DATA step or PROC GROOVY code to process the JSON results. Such methods get the job done (JSON is simply text, after all), but they aren't as robust as an official JSON parser. JSON is simple: it's a series of name-value pairs that represent an object in JavaScript. But these pairs can be nested within one another, so in order to parse the result you need to know about the object structure. A parser helps with the process, but you still need to know the semantics of any JSON response.

SAS 9.4 introduced PROC JSON, which allows you to create JSON output from a data set. But it wasn't until SAS 9.4 Maintenance 3 that we have a built-in method to parse JSON content. This method was added as a DS2 package: the JSON package.

I created an example of the method working -- using an API that powers our SAS Support Communities! The example queries communities.sas.com for the most recent posts to the SAS Programming category. Here's a small excerpt of the JSON response.

 "post_time": "2015-09-28T16:29:05+00:00",
  "views": {
  "count": 1
  },
  "subject": "Re: How to code for the consecutive values",
  "author": {
  "href": "/users/id/13884",
  "login": "ballardw"

Notice that some items, such as post_time, are simple one-level values. But other items, such as views or author, require a deeper dive to retrieve the value of interest ("count" for views, and "login" for author). The DS2 JSON parser can help you to navigate to those values without you needing to know how many braces or colons or commas are in your way.

Here is an example of the result: a series plot from PROC SGPLOT and a one-way frequency analysis from PROC FREQ. The program also produces a detailed listing of the messages, the topic content, and the datetime stamp.

series

boardfreq
This is my first real DS2 program, so I'm open to feedback. I already know of a couple of improvements I should make, but I want to share it now as I think it's good enough to help others who are looking to do something similar.

The program requires SAS 9.4 Maintenance 3. It also works fine in the most recent version of SAS University Edition (using SAS Studio 3.4). All of the code runs using just Base SAS procedures.

/* DS2 program that uses a REST-based API */
/* Uses http package for API calls       */
/* and the JSON package (new in 9.4m3)   */
/* to parse the result.                  */
proc ds2; 
  data messages (overwrite=yes);
    /* Global package references */
    dcl package json j();
 
    /* Keeping these variables for output */
    dcl double post_date having format datetime20.;
    dcl int views;
    dcl nvarchar(128) subject author board;
 
    /* these are temp variables */
    dcl varchar(65534) character set utf8 response;
    dcl int rc;
    drop response rc;
 
    method parseMessages();
      dcl int tokenType parseFlags;
      dcl nvarchar(128) token;
      rc=0;
      * iterate over all message entries;
      do while (rc=0);
        j.getNextToken( rc, token, tokenType, parseFlags);
 
        * subject line;
        if (token eq 'subject') then
          do;
            j.getNextToken( rc, token, tokenType, parseFlags);
            subject=token;
          end;
 
        * board URL, nested in an href label;
        if (token eq 'board') then
          do;
            do while (token ne 'href');
               j.getNextToken( rc, token, tokenType, parseFlags );
            end;
            j.getNextToken( rc, token, tokenType, parseFlags );
            board=token;
          end;
 
        * number of views (int), nested in a count label ;
        if (token eq 'views') then
          do;
            do while (token ne 'count');
               j.getNextToken( rc, token, tokenType, parseFlags );
            end;
            j.getNextToken( rc, token, tokenType, parseFlags );
            views=inputn(token,'5.');
          end;
 
        * date-time of message (input/convert to SAS date) ;
        * format from API: 2015-09-28T10:16:01+00:00 ;
        if (token eq 'post_time') then
          do;
            j.getNextToken( rc, token, tokenType, parseFlags );
            post_date=inputn(token,'anydtdtm26.');
          end;
 
        * user name of author, nested in a login label;
        if (token eq 'author') then
          do; 
            do while (token ne 'login');
               j.getNextToken( rc, token, tokenType, parseFlags );
            end;
            * get the author login (username) value;
            j.getNextToken( rc, token, tokenType, parseFlags );
            author=token;
            output;
          end;
      end;
      return;
    end;
 
    method init();
      dcl package http webQuery();
      dcl int rc tokenType parseFlags;
      dcl nvarchar(128) token;
      dcl integer i rc;
 
      /* create a GET call to the API                                         */
      /* 'sas_programming' covers all SAS programming topics from communities */
      webQuery.createGetMethod(
         'http://communities.sas.com/kntur85557/' || 
         'restapi/vc/categories/id/sas_programming/posts/recent' ||
         '?restapi.response_format=json' ||
         '&restapi.response_style=-types,-null&page_size=100');
      /* execute the GET */
      webQuery.executeMethod();
      /* retrieve the response body as a string */
      webQuery.getResponseBodyAsString(response, rc);
      rc = j.createParser( response );
      do while (rc = 0);
        j.getNextToken( rc, token, tokenType, parseFlags);
        if (token = 'message') then
          parseMessages();
      end;
    end;
 
  method term();
    rc = j.destroyParser();
  end;
 
  enddata;
run;
quit;
 
/* Add some basic reporting */
proc freq data=messages noprint;
    format post_date datetime11.;
    table post_date / out=message_times;
run;
 
ods graphics / width=2000 height=600;
title '100 recent message contributions in SAS Programming';
title2 'Time in GMT';
proc sgplot data=message_times;
    series x=post_date y=count;
    xaxis minor label='Messages';
    yaxis label='Time created' grid;
run;
 
title 'Board frequency for recent 100 messages';
proc freq data=messages order=freq;
    table board;
run;
 
title 'Detailed listing of messages';
proc print data=messages;
run;
 
title;

I also shared this program on the SAS Support Communities as a discussion topic. If you want to contribute to the effort, please leave me a reply with your suggestions and improvements!

tags: DS2, JSON, REST API, SAS 9.4

The post Using SAS DS2 to parse JSON appeared first on The SAS Dummy.

8月 032015
 

With apologies to this candy advertisement from the 1980s:

"Hey, you got your Lua in my SAS program."
"You got your SAS code in my Lua program!"

Announcer: "PROC LUA: Two great programming languages that program great together!"

What is Lua? It's an embeddable scripting language that is often used as a way to add user extensions to robust software applications. Lua has been embedded into SAS for some time already, as it's the basis for new ODS destinations like EXCEL and POWERPOINT. But SAS users haven't had a way to access it.

With SAS 9.4 Maintenance 3 (released July 2015), you can now run Lua code in the new LUA procedure. And from within that Lua code, you can exchange data with SAS and call SAS functions and submit SAS statements. (Running SAS within Lua within SAS -- it's just like Inception.)

Paul Tomas, the developer for PROC LUA, presented a demo of the feature and its usefulness in a recent SAS Tech Talk:


 
Paul also wrote a paper for SAS Global Forum 2015: Driving SAS with Lua.

Like many innovations that find their way into customer-facing features, this new item was added to help SAS R&D complete work for a SAS product (specifically, the new version of SAS Forecast Server). But the general technique was so useful that we decided to add it into Base SAS as a way for you to integrate Lua logic.

PROC LUA can be an alternative to the SAS macro language for injecting logical control into your SAS programs. For example, here's a sample program that generates a SAS data set only if the data set doesn't already exist.

proc lua ;
submit; 
 
-- example of logic control within LUA
if not sas.exists("work.sample") then
    print "Creating new WORK.SAMPLE"
	sas.submit [[
	  data work.sample;
	    set sashelp.class;
	  run;
	 ]]
   else print "WORK.SAMPLE already exists"
 end
 
endsubmit;
run;

First run:

NOTE: Lua initialized.
Creating new WORK.SAMPLE
    data work.sample;
      set sashelp.class;
    run;

And subsequent runs:

NOTE: Resuming Lua state from previous PROC LUA invocation.
WORK.SAMPLE already exists
NOTE: PROCEDURE LUA used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

Unlike other embedded languages (like PROC GROOVY), Lua runs in the same process as SAS -- and not in a separate virtual machine process like a Java VM. This makes it easy to exchange information such as data and macro variables between your SAS and Lua programming structures.

If you have SAS 9.4M3 and have time to play with PROC LUA, let us know what interesting applications you come up with!

tags: Lua, SAS 9.4

The post Using Lua within your SAS programs appeared first on The SAS Dummy.