SAS Environment Manager

7月 292020
 

Students in my classes often want to know about how to discover what's going on in their SAS environment. They get auditing questions from others in their organizations and don’t know how to find the answers. I honestly get a bit giddy when I can point them to SAS® software's Report Center, so it seems fitting to tout it now, as Friday is System Administrator Appreciation Day.

The Report Center, available since SAS 9.4M3, is a collection of stored processes that produce reports from data in the SAS Environment Manager Data Mart. These reports are a window into the performance and status of your SAS environment and its resources. They're samples of the types of reports you can produce using available metric data. You can also create your own reports to meet your individual requirements.

How Report Center works

The stored processes in the Report Center are created when you initialize SAS Environment Manager Extended Monitoring. However, the stored processes operate only on data that was stored in the SAS Environment Manager Data Mart by the APM or ACM ETL processes. Unless you initialize and enable one of those packages, no reports are produced.

The information in SAS Environment Manager Data Mart is the storage area for the Audit Performance Metrics (APM) and Agent Collected Metrics (ACM). APM scans the components in your SAS system for SAS server logs, SAS job logs, SAS Metadata and HTTP access logs; basically, everything SAS. The ACM collects information such as workload, CPU usage, and memory.

The current SAS Administration Fast Track course, which includes information about the Report Center, is taught on SAS 9.4M6. Before I decided to write this post, I searched and found others have provided information about the Report Center. There are several other blog posts and videos about it, so it surprises me how many I teach are unaware of the Report Center.

After generating reports in class, I have a whole new band of converts for the SAS Report Center. The great news is that it is never too late to initialize the Report Center in your SAS 9 environment.

Learn more about how to use the Report Center in the SAS Environment manager 2.5 User's Guide. Training for the SAS Report Center is found in the SAS® Platform Administration: Fast Track. Happy SysAdmin Day a couple of days early! I look forward to the opportunity to meet you in one of our upcoming classes!

TAKE THE CLASS | SAS® PLATFORM ADMINISTRATION: FAST TRACK

Report Center: SAS SysAdmin's secret weapon was published on SAS Users.

10月 192019
 

Newcomers to SAS Viya Administration may appreciate these tried-and-tested patterns for securing folders, and the content within them (reports, data plans, models etc.). Nothing too fancy today; if you are new to security model design in SAS Viya, this post is for you.

It presents five patterns for permissions on folders, reports, data plans and the like, and shows how you can combine them and apply them to design a simple but effective security model (or authorization model) for a folder structure.

SAS Viya has two authorization systems

If you are just starting work with permissions in SAS Viya, then you should be aware that there are TWO authorization systems in SAS Viya:

  • The general authorization system is used to secure (manage access to) folders, reports, data plans, models and other content stored in SAS Viya’s database (the SAS Infrastructure Data Platform, which uses PostgrSQL behind the scenes). It is also used to manage access to SAS Viya applications and some of their features.
  • The CAS authorization system is used to secure (manage access to) data, held in or accessed through SAS Cloud Analytics Services (CAS).

There is no overlap between the two systems. If you are interested in the permissions on a CAS library or table, you use the CAS authorization system. For everything else stored inside SAS Viya, it’s the general authorization system. Click the links above for the documentation. It’s worth mentioning that SAS Viya does make use of the host operating system’s filesystem, and can also access data in other databases or in SAS 9, each of which has its own authorization system too. These are therefore important too, but they usually take up less of a SAS Administrator’s attention.

SAS Viya has a built-in custom group called SAS Administrators, which is granted broad administrator access to much of the SAS Viya system out of the box. It is not a good idea to make users members of SAS Administrators unless they will bear at least joint responsibility for the health and stability of your SAS Viya deployment. Departmental ‘power users’ who just need read and write access to all your users’ content should not be made members of your SAS Administrators group! It’s better to make a content administrators group (or equivalent) and grant that group more access than you grant to most users.

Editing folder permissions in SAS Environment Manager

In the Content page in SAS Environment Manager (and some other places too), you can edit the authorization settings for folders, reports and other objects stored by SAS Viya. For most objects, e.g. reports, data plans, models etc, the Edit Authorization dialog looks like this. I’ve overlaid a blue shape, intended to emphasize that all four permissions apply to the object itself:

The Edit Authorization dialog for a folder has two more permissions (add and remove), and also has a second set of permissions. Those on the left apply to the folder itself, while those on the right, with ‘(convey)’ in their names, apply to the folder’s contents (technically, they apply to the folder’s ‘container’ and are conveyed to its contents). Again I’ve overlaid blue shapes to emphasize this:

This allows you to easily set different permissions on the folder itself from those you set on the folder’s contents: you can prevent users from renaming, deleting or moving a folder, but allow them to edit everything inside it. That’s nice!

When securing folders and the content inside them in SAS Viya, three basic guidelines to follow are:

  • Secure folders, not individual objects (reports etc.) wherever possible. It’s simpler to manage, and easier to understand later. If two reports in the same folder should have different permissions, they should probably be in different folders.
  • Grant permissions to groups, not individual users. It’s simpler to manage, and easier to understand later. If there isn’t a group that contains the set of users you need, either make a Custom Group, ask your Active Directory or LDAP administrator to make a new group for you, or consider using a larger existing group if that will work without granting too much access to someone who should not have it.
  • Avoid prohibits, especially ‘(convey)’-ed prohibits. Try to design your authorization scheme or security model so that you grant permissions wherever you need them, to give just enough access to content, and not too much. If a group of users should have access to a folder, and some of its contents, don’t grant Read (convey) permission on the parent folder, and try to prohibit Read permission on the subfolder(s) that the group of users should not be able to see. That would initially work, but it’s poor design and will likely become an inconvenience later. Instead, grant the group Read access, not Read (convey), on the folder, and then selectively grant that group Read and Read (convey) on the contents that they should be able to see.

These guidelines give rise to some recurring patterns of access. Note that you won’t find these patterns anywhere in the software: there is no screen, or button, or command for defining and applying them. They are design concepts only. This post’s five simple general authorization patterns are as follows (click any image to enlarge it):

No Access

The default. SAS Viya does not give user access to anything unless it is directly or indirectly granted.

This pattern is of course only effective if the user or group does not have access conveyed from a parent folder. So long as it does not, your masterly inactivity on this folder’s permissions ensures it too is not accessible to this user or group. If the folder does have access conveyed for the user or group in question from higher up the folder hierarchy, remove those conveyed permissions, and grant the permissions you require (whether conveyed or not) more selectively to subfolders or to a smaller group, so that the group in question does not get access they should not get.

<Group> Read

Grants a group permission to see this object only.

When a group should have read access to an object (ideally a folder), but not the objects’ contents, apply this pattern for that group. To apply it, grant a specific group Read permission on this folder, on the left-hand side of the authorization grid (or if you are using a programmatic method to set permissions, on the object URI), and grant the group nothing on the right-hand side of the grid (or on the container URI).

The 'Group' in angled brackets is written that way because that you might use variants of this pattern for several different groups: you may use an HR Read pattern, a Sales Read pattern, or a North American Finance Modelers Read pattern, as demanded by your requirements. Substitute the name of an actual group to turn the generalised, abstract pattern into a specific concrete pattern.

<Group> Read Convey

Grants a group permission to see this object and everything inside it.

To apply this pattern, grant a specific group Read permission on this folder, on the left-hand side of the authorization grid (or if you are using a programmatic method to set permissions, on the object URI), and also grant it the Read permission on the right-hand side of the grid (or on the container URI).

<Group> Edit Contents

Grants a group permission to make changes to all of the content of this folder. When you grant permissions in the Group Edit Contents pattern, you will usually also grant the permissions in the Group Read Convey pattern too: users who edit things need to see them.

For this pattern, grant the group Add and Remove on the object itself, and grant it Update, Delete, Add and Remove on the object’s container. These permissions allow the group to add and remove objects from the folder, to modify the objects inside the folder, and to add and remove objects from any subfolders.

Usually, you wouldn’t apply this pattern on its own. If a group is allowed to edit the contents of a folder, they usually need to be able to see the folder too. So, apply this Group Edit Contents pattern plus the Group Read Convey pattern together, for the same group on the same object:

<Group> Secure

Grants a group permission to add, remove, or change permissions on this object.

To apply this pattern, grant a specific group Secure on this folder, and Secure (convey) on the container.

When you grant permissions in the Group Secure pattern, you will usually also grant the permissions in the Group Read Convey and Group Edit Contents patterns too: users who manage objects’ permissions are generally super-users, content administrators or similar, and need to be able to see and edit the things they are allowed to secure:

Using the patterns

We have seen five simple regularly-occurring general authorization system permissions patterns, and a couple of ways in which some of them are usually combined. You may have instances of each of these patterns for each of several different groups in a security model design. Let’s see how they can be applied to a fictitious folder structure for the made-up organization we use in some of our GEL Administration workshops, GELCorp:

You may notice that the symbols in this table’s cells correspond to the symbols for the patterns presented above. The white symbols show patterns of permissions that should be granted to the group in the column header, on the folder in the row header. The grey symbols show inherited permissions, to remind us that they are in effect on a folder, but do not need to be directly granted on it again.

I’d encourage you to attend one our GEL SAS Viya Administration workshops, or the Securing SAS Viya Deployments workshop both available in the VLE or as a face-to-face workshop, for a fuller explanation of why we’ve applied the permissions patterns in this way. This security model design is certainly not the only way you could choose to secure GELCorp’s folders. But I expect you can see, at a glance, how the folders are intended to be secured, even if it is not always obvious from this diagram why. Plus, if you were to follow this design for each group and each folder where a symbol appears, you would get exactly the same set of permissions we apply in our workshop.

Hopefully you would agree that this is quite an easy way to represent and the basic permissions design for a set of folders, for multiple groups!

Simple general authorization patterns was published on SAS Users.

10月 152019
 

In a previous post, I discussed using logs to troubleshoot problems in your Viya environment. In this post, I will look at some additional ways to troubleshoot using some of the tools provided by the Viya Operations Infrastructure. With applications, servers and numerous micro-services all working together and generating their own logs in Viya, it can be difficult to find relevant logs. In order to manage the large number of logs and to enable you to locate messages of interest, the operations infrastructure provides components to collect and store log messages.

The collection process is illustrated in the diagram below.

Co-ordinated by the operations infrastructure:

  • sas-watch log continuously collects and sends log messages to the RabbitMQ exchange
  • sas-stream pulls the messages from RabbitMQ and writes them to disk as a tab-separated value (TSV) file
  • Every five minutes, the sas-ops-agentsrv runs the DatamartEtl task to extract log messages from the TSV file and load them into the VIYALOGS CAS-indexed search table

SAS Environment Manager uses the information in the VIYALOGS table and the VIYALOGS_SOURCES tables to display log messages and graphs that contain the frequency and trends of messages. The SAS Environment Manager LOG’s interface makes it really easy to search and analyze log messages. Using the interface, you can view, subset and search logs. The interface has the filtering capabilities on the left hand side and displays the messages on the right. By default, the filter is set to display all messages from all applications and services from the last 30 minutes.

You can modify the filter to extend or shorten the timeframe, subset the level of messages displayed or the source (service/application) that the messages are coming from. You can also search for any text within a message.

Many administrators would prefer a command-line interface, and the good news is there is one.

sas-ops is a command-line interface which allows for the monitoring of the operational infrastructure in a SAS Viya deployment environment.

I have found the sas-ops log command very useful to troubleshoot problems. The sas-ops log command can be used to stream log messages that are generated by SAS Viya applications and services. The messages can be streamed to a terminal window or piped to a file. The sas-ops logs command is located at /opt/sas/viya/home/bin and can be run from any machine in a Viya environment that is included in the CommandLine.

When would you use sas-ops logs to stream log messages? Some potential scenarios are to:

  • troubleshoot a poorly performing report or analysis
  • debug problems in the environment such as logon issues
  • monitor access to resources

In these cases, using sas-ops logs you can stream the log messages from all services to a single file or terminal.

In its simplest form, the command live streams all log messages from a Viya environment to the terminal. Selecting CTRC+C will stop the streaming.

./sas-ops logs

Partial output from the stream is shown below.

If you want to save the output, you can redirect the stream to a file.

./sas-ops logs &gt; /tmp/mylog.log

You can get more creative and achieve more complex tasks. You can change the format of the message output using –format. For example, to create a file with json which could be read by another process use:

./sas-ops logs –format pretty &gt; mylogs.json

You can also:

  • stream messages for just a specific Viya service
  • filter logs messages by text in a regular expression
  • stream for a specific duration

The duration is specified using the format 0h0m0s0ms, but you can also use individual parts of the specification, for example 30s for 30 seconds or 5m for 5 minutes.

Consider the situation where we want to monitor access to a particular CAS table over a specific period of time. The command below will output to a file all messages that contain the table name HR_SUMMARY for a period of 5 minutes.

./sas-ops logs –match HR_SUMMARY –timeout 5m &gt; /tmp/hr_summary_access.log

The output shows all the CAS actions that were performed on the table during the time period.

You can subset the stream to one service.

Consider a case where a user is having an issue logging in and you suspect you have an issue with the LDAP setup. To check the problem, you can firstly enable DEBUG logging on com.sas.identities. Then stream the log messages from the identities service.

./sas-ops logs –format pretty –source identities &gt; logonerrors.json

Viewing the output shows that there is something wrong with the LDAP query.

I think you will agree that sas-ops logs is a very useful tool for monitoring and troubleshooting issues in a Viya environment. For more information, check out the following resources:

I would like to thank Bryan Ellington for his helpful input with this post.

Capturing log messages from Viya deployments was published on SAS Users.

6月 142017
 

In SAS Viya 3.2, the Self-Service Import provides a mechanism for a user to import (copy) data into the SAS Cloud Analytic Services (CAS) environment. The data is copied as a .sashdat file into the selected CAS Library location when it is imported.  Self-Service Import data can only be imported into CAS libraries of type PATH, HDFS, or DNFS.

The Self-Service Import functionality is available in the following applications:

  • SAS Visual Data Builder
  • SAS Visual Analytics
  • SAS Environment Manager – Data

To have access to Self-Service Import, the end user must be granted the Read permission on the /casManagement_capabilities/importData object URI in the Security ⇨ Rules area of SAS Environment Manager.

Self-Service Import supports importing data to CAS from Local, Server, and Social Media sources.

Self-Service Import in SAS Viya

SAS Viya 3.2: Self-Service Import

Local

Local file data can be imported from Microsoft Excel (.XLSX or .XLS), text file (.TXT or .CSV), the clipboard, or a SAS Data Set (SASHDAT or SAS7BDAT). The file(s) must exist on a file system available to your PC.

Server

After providing the appropriate server connection information, a table from LASR or select database types can be imported. The currently supported database types are:  Oracle, Teradata, Hadoop, PostgreSQL, and Impala. The Server selections displayed are dependent on your licensing and configuration.

Social Media

After authentication with the social media provider (Twitter, Facebook, Google Analytics, or YouTube), data can be imported through the social media provider’s public API. Access to these APIs is subject to the social media provider’s applicable license terms, terms of use, and other usage terms and policies.

Currently, there is a size limit for file imports that is set on the CAS Management service Configuration screen in SAS Environment Manager. The default size is 4GB. The local file importer has a 4GB limit because that is what the smallest size limit browser (Internet Explorer) is restricted to; however, Chrome and other browsers will allow larger file sizes, which is why there is a property that allows an Admin to set a higher limit. A modification to the max-file-size property requires a restart of the casManagement service.

Social Media and DBMS importers have no explicit limits. However, there is a limitation of the disk size of where casManagement is running because the uploaded file gets written to a temporary file on the server relative to casManagement.

For more information refer to the Self-Service Import section of the The Self-Service Import in SAS Viya 3.2 was published on SAS Users.

12月 212016
 

The report-ready SAS Environment Manager Data Mart has been an invaluable addition to SAS 9.4 for SAS administrators. The data mart tables are created and maintained by the SAS Environment Manager Service Architecture Framework and provide a source of data for out-of-the box reports as well as custom reports that any SAS administrator can easily create. As you can imagine, the size of the tables in the data mart can grow quite large over time so balancing the desired time span of reporting and the size of the tables on disk requires some thought. The good news: SAS 9.4 M4 has made that job even easier.

The Environment Manager Data Mart (EVDM) has always provided a configuration setting to determine how many days of resource records to keep in the data mart tables. You can see below that in a fresh SAS 9.4 M4 installation, the default setting for “Number of Days of Resource Records in Data Mart” is set to 60 days. This means that EVDM data records older than 60 days are deleted from tables whenever the data mart ETL process executes.

EV Data Mart Tables in 9.4M4

The space required to house the Environment Manager Data Mart is split across three primary areas.

  • The ACM library tables contain system level information
  • The APM library tables contain audit and performance data culled from SAS logs
  • The KITS library tables contains miscellaneous tables created by data mart kits that collect specialty information about HTTP access, SAS data set access, and such.

Prior to SAS 9.4M4, the ACM and APM libraries duly archived data according to the “Number of Days of Resource Records in Data Mart” setting, but the KITS library did not. For most of the KITS tables this is not such a big deal but for some deployments, the HTTPACCESS table in the KITS library can grow quite large. For administrators who have enabled the VA feed for the Service Architecture Framework, the size of the HTTPACCESS table directly impacts the time it takes to autoload the results of each refresh of the data mart, as well as the amount of memory consumed by the LASR Server used for the Environment Manager Data Mart LASR library.

So what is the big change for SAS 9.4 M4?

The KITS library now respects the “Number of Days of Resource Records in Data Mart” setting and removes data older than the threshold.  If you are a SAS administrator, you can now forget about having to separately manage the KITS library which should simplify space management.

SAS administrators may need to adjust the “Number of Days of Resource Records in Data Mart” setting to strike a balance between the date range requirements for reporting and the amount of disk space they have available for storing the EVDM tables.  With SAS 9.4 M4, however, administrators can rest assured that all EVDM tables will self-manage according to their wishes.

More on the Service Architecture Framework.

tags: SAS 9.4, SAS Administrators, SAS Environment Manager, SAS Professional Services

Easier Space Management for EV Data Mart Tables in 9.4M4 was published on SAS Users.

10月 062016
 

One very useful type of auditing for a SAS administrator is to have summary data about the availability and performance of various resources (platforms, servers, services) from the 30,000-foot view.  Using SAS Environment Manager, it's easy to go in and look at the availability of any one resource over various time spans--for the past few hours, past day, past week, or past month and more.  This is a very powerful way to summarize how much of the time a given resource was "up and responsive."

auditing-sas-server-availability

However, there's no way to see that type of information, or even a summary of that information, for all your servers at once.  A typical deployment will have anywhere from 10 to 50 or more different servers, and to view the availability of them all, over an extended period, you would have to visit and drill down each resource, one at a time.

To help answer this problem, I've developed a simple report that summarizes the availability for all servers.  It uses two data sets that are automatically generated as part of the SAS Environment Manager Data Mart - availability and resourceinventory. It doesn't provide  the hour-by-hour information like the Monitoring interface of SAS Environment Manager does, but it gives you  a percent of time available, for each server, for either the past day, week,  month, or even quarter, in one summary report (if you have the data for it).  For a production environment, this could provide helpful "big-picture" information on availability.

Building the report will involve copying an existing report, used as a template, modifying a bit of the metadata for that report, and copying the SAS code that generates the report.  The report uses a stored process to do the work.

Here's how to create the report.

1. Copy the SAS code provided here and save it in a local file:

If you are working on a remote machine, such as through Remote Desktop or mobaXterm, then upload the file to the machine running the metadata server so it will be available to copy into the report metadata.

2. Log into SAS Management Console as sasadm@saspw, the SAS administrator.

3. Select the Folders tab (upper left), right mouse on the Shared Data folder, select New->Folder, and name it Custom Reports:

auditing-sas-server-availability02

4. Navigate to the folder Products->SAS Environment Manager->Custom and locate the existing example report called Example 2 Call EV Macros with prompts.

auditing-sas-server-availability03

5. Copy the example report into the new Custom Reports folder created in Step 3 above, then use the right mouse menu to rename it to Availability Report.  When complete, you will see the new folder and report inside it:

auditing-sas-server-availability04

6. Select the new Availability Report, right mouse, select Properties.

7. On the General tab, specify Name, Description, and Keywords as shown:

auditing-sas-server-availability05

It's especially important to have the keywords "Environment Manager" in the Keywords field, so that the report will appear in the Report Center of Environment Manager.

8. On the Execution tab, select your Application server, and below, specify Store source code in metadata.  Then click the Edit Source Code... button:

auditing-sas-server-availability06

9. Copy/paste the SAS code (Step 1 above) into the source code window and click OK to save it.

auditing-sas-server-availability07

10.  On the Parameters tab, you'll see Titles and Footnotes, and Output Formats and Debugging.  Select and Delete the Titles and Footnotes group; leave the Output Formats and Debugging group as is.

auditing-sas-server-availability08

11. Select General, then click the New Group... button (right side), and specify:

Group type:  Standard group

Displayed text: Time Period

Click OK button

auditing-sas-server-availability09

12. Select the new group, Time Period.  Click the New Prompt button.  Type the string "TimePeriod" (no space) in the Name field and “Time Period” in the Displayed Text field.  Then click OK.

auditing-sas-server-availability10

13.  Select the new prompt TimePeriod, click the Edit button on the right side.  Select the Prompt Type and Values tab at the top.  On the Edit Prompt screen, specify:

Prompt type:   Text

Method for populating prompt:  User selects values from a static list

Number of values: Single value

Use the Add button (right side) to insert the following four choices shown below:

auditing-sas-server-availability11

Set the "week" value as the default as shown.   Click the OK button.  Now, select the new Time Period prompt and click the Move Up button (right side), so that the Time Period prompt appears first:

auditing-sas-server-availability12

If desired, you can try the Test Prompts button to make sure your prompts are correct.  Click OK again to save all changes.

14. Test the new report:
Log into SAS Environment Manager as an administrator and go to the Report Center. Open the Shared Data / Custom Reports folder and find the new report "Availability Report."  (If you were already logged into Environment Manager, you have to log out/log in again to pick up the new metadata.)

auditing-sas-server-availability13

Notice that you have the custom prompt for Time Period , where you can choose either "day," "week," "month" or "quarter" for the time window, and the standard prompt for Outputting Styles and Logging and Debugging settings.   Click the Run button at the bottom, and you should see your new report, similar to this:  (Yellow indicates any percent less than 100.)

auditing-sas-server-availability14

You can install this report for any site with the SAS Environment Manager Data Mart active and have it running in just a few minutes.  You could pull the SAS code from it and create your own scheduled run, weekly or monthly, as a means to help to evaluate a deployment for reliability.

A few cautions are in order:

  • You need to have at least some data in your data mart in order to run this report. If you don't have enough data to cover the time period that you request, you will get an error message.
  • The data this report provides is summary level, so if you find one or more servers
    that have been down a significant amount of time, the next step would be to use the SAS Environment Manager to focus in on those troubled resources and examine exactly WHEN they were down, for exactly how long, how many times they were down, and/or look at other metrics on those resources as a way to ascertain causes.
  • This is a simple program without much error checking.  However, you can specify in the parameters to see the SAS log, which should enable you to debug it should issues occur.  And of course feel free to enhance it or add to it!

One easy check you can do by hand is to verify the existence of the data sets being used, and that the libref used, ACM, has been defined.  This is what they looked like on my compute machine, where the SAS Environment Manager Service Architecture was installed:

auditing-sas-server-availability15

auditing-sas-server-availability16

For additional insight on how you can use SAS Environment Manger and the report center to analyze the "big picture" on a SAS deployment's availability, check out this YouTube video.

SAS code referenced above.

/*********************************************************************/
/*   Availability.sas       SAS 9.4   M3                             */
/*      Runs report in SAS EV (Report Center) on availability of     */
/*      server resources.  Requires macro variable to be defined     */
/*      TimePeriod, with value of (dtday, dtweek, dtmonth, or dtqtr) */
/*      Requires data sets:  acm.resourceinventory                   */
/*                           acm.availability                        */
/*            from Service Architecture Framework datamart           */
/*                          Dave Naden August 2016                   */
/*********************************************************************/
 
*ProcessBody;
 
%global _DEBUG
        _EVDEBUG
        _EVODSDEST
        _EVODSSTYLE
        _EVSHOWPARM
        FNOTE1
        FNOTE2
        FNOTE3
        TITLE1
        TITLE2;
 
 
%evdm_kickstart();
%include "&amp;SASEVCONFIGDIR/Datamart/EVDMAUTOEXEC_STP.sas";
 
ods _ALL_ CLOSE;
 
%evodssetopt(evdest=&amp;_evodsdest,style=&amp;_evodsstyle);
%evgopts;
 
%evspinit;
 
%ev_promptselects(varname=_debug);
 
%let evbegin=%ev_startdate(%sysfunc(date()),-10);
%let evend=%sysfunc(datetime(),datetime.);
 
%let period=&amp;TimePeriod;     /* can be dtday, dtweek, dtmonth, or dtqtr */
 
%STPBEGIN;
%macro available;
/* get the resource IDs for servers only  */
data servers;
   set acm.resourceinventory(keep=id name type invLevel);
      if (invlevel = "SERVER");
     /* remove a few servers that are not really servers  */
      realserver = 1;
      if (index(type,"Directory") &gt; 0) then realserver = 0;
      if (index(type,"Data Mart") &gt; 0) then realserver = 0;
      if (index(type,"Server Context") &gt; 0) then realserver = 0;
      if (index(type,"System Info") &gt; 0) then realserver = 0;
      if (realserver) then output;
 
      rename id = resource_id;
 
 proc sort data=servers;
    by resource_id;
 
/* get availability data, prepare for merge */
 proc sort data = acm.availability(keep=resource_id name avail starttime endtime minutes)
    out=avail;
    by resource_id;
 
 /* keep data only if it's server data, and if there's any records  */
 /*  in the availability data set   */
 data avail;
    merge servers(in=a)
          avail;
       by resource_id;
       format now datetime24. ;
      if (a) and (avail ne .) ;
 
 /* get maximum (most recent) endtime  */
 proc summary data=avail;
    output out=timewindow(rename=(endtime=maxendtime)) max=; 
    var endtime;
 
 /* Get earliest start time for this data set  */
 proc summary data=avail;
    output out=earliesttime(rename=(starttime=earliest)) min=;
    var starttime; 
 
%let enoughdata=1;  
 
 /* establish the time window for report requested, and check whether  */
 /* data goes back far enough to run the report.  If not, set flag     */
 data timewindow;
    set timewindow ;
    set earliesttime;
       period = symget('period');
       begintime = intnx(period,maxendtime,-1,'S');
       format begintime datetime24.  ;
       totalminutes = intck('DTMINUTES',begintime,maxendtime,'C');
       if (earliest &gt; begintime) then call symput('enoughdata','0');
run; 
/* subset data set to include only records within that time window requested  */
%if &amp;enoughdata %then %do;
data avail;
   if (_N_ = 1) then set timewindow;
   set avail;
      if (endtime &lt; begintime) then delete;
 
 /* aggregate to resource level (server), calculate percent available */
 data avail(keep=name type resource_id minutesdown percentup  );
    set avail;
       by resource_id;
       retain minutesdown;
          if first.resource_id then minutesdown=0;
          if (avail &lt; 1) then do;
             if (starttime &lt; begintime) then do;
             /* get number of minutes between begintime and endtime */
                minutes = intck('DTMINUTES',begintime,endtime,'C'); 
             end; 
             minutesdown = minutesdown + minutes;
          end;   
          if last.resource_id then do;
             if (minutesdown = 0) then percentup = 100;
             else percentup = ((totalminutes - minutesdown) / totalminutes) * 100;
             output;
          end;
   call symput("begintime",put(begintime,datetime24.));
   call symput("maxendtime", put(maxendtime,datetime24.)) ;
   call symput ("currentdatetime",put(datetime(),datetime24.)) ;
   %let period = %substr(&amp;period,3);        
 
proc sort data=avail;
   by percentup;
 
 
proc report data=avail center split="*";
title "Server-type resources, showing percent up time up, past &amp;period only";
title2 "Date Interval used: &amp;begintime to &amp;maxendtime "; 
footnote "Date of Report:  &amp;currentdatetime ";
   column name type minutesdown percentup; 
   compute percentup;
      if percentup.sum
tags: SAS Administrators, SAS Environment Manager, SAS Professional Services

Auditing SAS server availability from 30,000 feet was published on SAS Users.

8月 022016
 

One of the jobs of SAS Administrators is keeping the SAS license current.  In the past, all you needed to do was update the license for Foundation SAS and you were done. This task can be performed by selecting the Renew SAS Software option in the SAS Deployment Manager.

More recently, many SAS solutions require an additional step which updates the license information in metadata. The license information is stored in metadata so that middle-tier applications can access it in order to check whether the license is valid. Not all solutions require that the SAS Installation Data file (SID) file be stored in metadata, however the list of solutions that do require it is growing and includes SAS Visual Analytics. For a full list you can check this SASNOTE. To update the license information in metadata, run the SAS Deployment Manager and select Update SID File in Metadata.

Recently, I performed a license renewal for a Visual Analytics environment. A couple of days later it occurred to me that I might not have performed the update of the SID file in metadata. That prompted the obvious question: how do I check the status of my license file in metadata?

To check the status of a SAS Foundation license you can use PROC setinit. PROC setinit will return the details of the SAS license in the SAS log.

proc setinit;run;

steps to update your SAS License

The above output of PROC setinit shows the:

  • Expiration Date as 25MAY2017
  • Grace Period ends on 09JUL2917
  • Warning Period ends on 04SEP2017

This indicates that the software expires on 25MAY2017, however nothing will happen during the Grace Period. During the Warning Period messages in the SAS log will warn the user that the software is expiring. When the Warning Period ends on 04SEP2017 the SAS Software will stop functioning. PROC setinit is only checking the status of the Foundation SAS license, not the license in metadata.

If the foundation license is up-to-date but the license stored in metadata is expired the web applications will not work. It turns out SAS Environment Manager will also monitor the status of the SAS license. But is it the Foundation license or the license stored in metadata?

To see the status of the license in SAS Environment Manager, select Resources then select Browse > Platforms > SAS 9.4 Application Server Tier. The interface displays:

  • Days Until License Expiration:  the number of days until the license expires.
  • Days Until License Termination: the number of days until the software stops working.
  • Days Until License Termination Warning: the number of days until the Grace period.

steps to update your SAS License

Some testing revealed that Environment Manager is monitoring not the status of the foundation license but the status of the license in metadata. This is an important point, because as we noted earlier not all SAS solutions require the SID to be updated in metadata. Since Environment Manager monitors the license by checking the status of the SID file in metadata, administrators are recommended, as a best practice, to always update the SID file in metadata.

Environment manager with Service Architecture configured also will generate events that warn of license termination when the license termination date is within a month.

In addition, as of SAS 9.4 M3, SAS Management Console has an option to View metadata setinit details. To access this functionality you must be a member of the SAS Administrators Group or the Management Console: Advanced Role.

To check on a SID file in metadata open SAS Management Console and in the plug-ins tab:

1.     Expand Metadata Manager

2.     Select Metadata Utilities

3.     Right- click and select View metadata setinit details

steps to update your SAS License

Selecting the option gives details of the current SID file in metadata, with similar information as PROC setinit displays including the expiration date, the grace period and the warning period.  In addition it displays the date the SID file was last updated in metadata.

steps to update your SAS License

The takeaway: to fully renew SAS software, and ensure that SAS Environment Manager has the correct date for its metrics on license expiration, always use SAS Deployment Manager to both Update the SAS License, AND Update the SID File in Metadata.

To check if your SAS Deployment license has been fully updated, do the following:

1.     Run PROC setinit to view the status of the SAS Foundation license.

2.     Use SAS Management Console or SAS Environment Manager to check if the SID file has been updated in metadata.

For more information on this topic see the video, “Use SAS Environment Manager to Get SAS License Expiration Notice” and additional resources below:

 

SAS® Deployment Wizard and SAS® Deployment Manager 9.4:User’s Guide: Update SID File in Metadata
SAS® Deployment Wizard and SAS® Deployment Manager 9.4:User’s Guide: Renew SAS Software
SAS(R) 9.4 Intelligence Platform: System Administration Guide: Managing Setinit (License) Information in Metadata
SAS® Environment Manager 2.5 User’s Guide

tags: configuration, SAS Administrators, SAS architecture, SAS Environment Manager, SAS Professional Services

Two steps to update your SAS License and check if it is updated was published on SAS Users.

6月 142016
 

As an addendum to my previous two blogs on using the SAS Environment Manager Report Center, this blog illustrates further tips and tricks you can use to help make the creation of your custom reports easier.

The Ad-Hoc Reporting section of the Report Center is specifically designed to provide a “testing ground” for reports you may want to try. It can be your most useful tool in designing and creating the reports you want. Notice that the first four “reports” are most useful if you have a good idea of what content you want your report to have; they present  specific report content, and it’s easy to see what type of data these reports would  contain, just based on the titles or names shown.

Auditing using the SAS Environment Manager Report Center

Selecting any of these four reports will generate a list of all possible reports in that category. From here, you can select the one(s) you want. Here are a few screenshots to illustrate:

APM Metadata Audit: If you make this selection in the Ad Hoc Reporting section, you will be presented with a choice of all reports in that category, under the Report Name field dropdown list in the right-hand panel. Notice that most of these report selections correspond to finished reports already created in the main report section, marked on the left below. If you just run this Ad Hoc report with all the defaults, you’ll get the same report as the one listed in the Nightly Reports >> Audit Reports section. However, you can test some additional parameters using the Ad Hoc version’s menu shown.

Auditing Using the SAS Environment Manager Report Center02

You will find similar choices for all four report groups in that section of the Ad Hoc Reporting. Thus, you have a way to tinker with many of the other standard reports, and try various modifications to them. For example, if you are interested in any of the “ARM Performance Reports,” you can open the ARM Performance section in the Ad Hoc Reporting section to get a set of reports that (mostly) mirror those reports:Auditing Using the SAS Environment Manager Report Center03

This allows you to explore different parameter options beyond what’s provided by the standard reports, such as chart type, number of time bins, and subgrouping time periods.

A different approach is to explore your data using any of the remaining Ad Hoc reports that are organized around the type of output plot you want. As discussed in the previous blog, the name of the Ad Hoc report shows the type of plot generated based on the SAS PROC name. These reports allow you to select a dataset to explore from any of the following three built-in DataMart libraries:  ACM, ARTIFACT, or KITS.  The Ad Hoc reports that can be used in this way are circled below:

Auditing Using the SAS Environment Manager Report Center04

For example, if I want a basic chart, and I want to use the dataset ARTIFACT.AUDITACCESSC, then I can select the Ad Hoc report called “evChart TimeSeq Explorer,” choose my data set from the dropdown list, and then try varying any parameters provided in the interface to explore and come up with many variations and charts based on those two choices. Notice that with this group of Ad Hoc reports, there are more parameter choices than with other reports:

You can select many reporting options, such as multiple variables, type of chart, class variables, break variables, subgroupings, statistics, as well as the more standard subsetting and output format options. You can generate literally hundreds of different reports using varying parameter selections using these Ad Hoc Reports.

As I mentioned in a previous blog in this series, if you want to create a custom report for production that you will run many times, you will want to make a permanent, new copy of the stored process being called, with the correct parameters specified on the reporting macro being called.  I outlined the steps to do this in the previous blog: Creating Custom Reports in the Report Center.

The way to do this is to model your report on an existing report, then make modifications to it. You will want to know the following about the report you are using as a model:

  • What stored process is being used?
  • What reporting macro(s) are being called?
  • What parameters are being specified to the reporting macro?
  • What data set from the Data Mart is being used?

The best way to answer these questions is by using the debugging facility that is built into the report center. All reports provide the following debugging options:

  • Debug Level
  • Session Information: Log
  • Macro Invocation Parameters

Auditing Using the SAS Environment Manager Report Center05

By browsing the SAS log and checking the called reporting macro and its parameters, you can quickly answer the four questions listed above.

If you know what data set you want to use but need to explore that data set further, there are two approaches:

  1. Try using the Ad-Hoc Reporting section as described above to explore the data sets.
  2. Use another data browsing tool–I’ve found that SAS Studio is perfect for this task.

When the Service Architecture Framework is initialized, the following data libraries are added to metadata automatically:

  • SASEV – ACM
  • SASEV – ARTIFACT
  • SASEV – EVDM
  • SASEV – KITS

For these cases you can just browse data directly using SAS Studio.  For the other Data Mart libraries, (EVDB, EVDMRAW), you will have to first define those libnames and register the tables in metadata. Once this is done you can explore the data sets in exactly the same way.

tags: SAS Administrators, SAS Environment Manager, SAS Professional Services

Auditing using the SAS Environment Manager Report Center– tips and tricks was published on SAS Users.

5月 182016
 

EnvironmentManagerMy previous blog discussed the SAS Environment Manager Report Center, and talked about its organization, and how to start using some of the prompts to get the reports you want. The next step is to learn to use some of the example reports provided to help you design your own, production-level reports.

It’s helpful to distinguish ad-hoc, or exploratory reporting, vs. standardized, production-style reports, which might be run daily, weekly, or any other number of times. You can experiment with many ad-hoc reports and create hundreds of variations and different reports “on the fly,” just by manipulating the various parameters provided within the Report Center interface. Perhaps more importantly, you can use knowledge gained from this process to design more permanent production reports, using some of the provided examples as “templates.”

First, you need to understand the basic structure: All the SAS Environment Manager Reports are SAS Stored Processes available from the Stored Process web application. Most of them accept user prompts that specify many details about the data to include and the way the output should appear. Then, the stored process passes these parameters to a small set of reporting macros, which perform the data manipulations and produce the actual report using well-known SAS Procedures.

In addition to all the pre-defined reports, there are two sets of example reports specifically intended to be used as templates from which you can develop your own custom report. Often, this only requires a few changes to parameters being passed to the reporting macro. These example reports are located here:

Under Products>SAS Environment Manager>Custom
Under System>Applications>SAS Environment Manager>Ad Hoc Reporting

Creating custom reports from the Data Mart01

In the first set, the Example2 Call EV Macros with prompts is likely of most interest. This report demonstrates how to call two of the most useful reporting macros:

%evtmpplate creates a wide variety of plots using PROC TEMPLATE
%evtabrpt  creates a wide variety of data tables using PROC TABULATE

The second set in Ad Hoc Reporting has a series of reports that use many additional reporting macros. Notice that many of the reports include the reporting macro name they showcase, which matches the corresponding stored process filename, in the report name itself. Thus you can quickly locate the most important files being used in the running of the report based on the report’s name. Here’s a table showing the most useful reporting macros:

Creating custom reports from the Data Mart02

Armed with this information, here’s the approach to take to create your own custom reports in the Report Center:

  1. Decide what you want to report on, and what general type of report you want to create (table, crosstab, bar chart, line graph, heat map, etc.)
  2. Try to find existing reports in the Report Center that may already do what you want, or are at least based on the data that you want to use
  3. Determine which Data Mart data set you want to report from, based on Steps 1 and 2
  4. If none of the existing reports provide what you want, try experimenting with all the parameters available in the Ad-Hoc reports section, (ACM, ARTIFACT, KITS libraries only) using the report type you want (plot, chart, tabulate, etc.) to determine the example report to try.  Most of these reports allow you a wide variety of choices, including:
    o  data set
    o  main analysis variables
    o  class variables, or BY variables
    o  type of plot, plus many options and details about the plots
    o  data subsetting by variable (WHERE clause), dates
    o  output formatting
    o  titles/footnotes/ODS style, etc.
  5. When running the Ad Hoc reports, first specify the data set and analysis variable(s) that you want, and then experiment with various other parameters. Open the file containing the macro code that you are using–the reporting macros are fully documented with all options and parameters accepted. Use the table above to help you find the correct file(s).
  6. Once you get the report that you want from the Ad Hoc section, then you can use the information you’ve gained to create a more permanent version. Here’s an overview of the steps to create a permanent custom report:

Caution before proceeding: Be careful not to modify any of the stored process files that are part of the Report Center. If you want to modify one of the stored processes, first make a copy and give it a new name. Then simply refer to the new name in that stored process’ metadata and modify the copy.  Or, you can put the stored process code directly in metadata instead of a file, in which case you don’t have to alter any Report Center files.

  1. Using SAS MC, copy the existing Ad Hoc report to a new metadata location, and rename it.
  2. Make a copy of the stored process file called by the report, and rename it.
  3. Edit the new (copied, renamed) stored process code, and you will see the reporting macro(s) being called.
  4. Modify the parameters on any called reporting macros with the new parameters you want, based on what you learned in the previous steps.  For more guidance, open the *.sas file containing the reporting macro, and follow the detailed documentation that’s included with each macro.
  5. Chances are that it will require a few tries before you get it exactly right, but it’s nice to see how much of the work has already been done for you with the existing examples.

More information on the steps to add a report to Report Center.

 

tags: SAS Administrators, SAS Environment Manager, SAS Professional Services

Creating custom reports from the Data Mart was published on SAS Users.

3月 302016
 

EnvironmentManagerThe SAS Environment Manager Report Center is a set of SAS stored processes and SAS macros which leverage the SAS Data Mart for the purpose of monitoring and auditing a SAS installation. Full documentation on the structure and functioning of the Data Mart can be found in the SAS Environment Manager 2.5 User’s Guide. This blog discusses functionality available as of the SAS 9.4 M3 release.

The intent of the Report Center is to provide most of the reporting that a SAS administrator might need “out of the box.” However, there’s lots of customization possible without doing any coding at all. You do, however, need to know something about your data, and, it helps if you’ve also got some knowledge of SAS reporting and graphing procedures. Those procedures would include PROC TABULATE, PROC TEMPLATE (for plots), PROC GPLOT, and PROC REPORT. This blog will explore a few things I’ve discovered about using the parameters to produce customized reports. In later blogs I’ll discuss using some of the reporting templates provided.

The first step is get a feel for the organization of the report center. Here’s a table to summarize what’s in it:

Opening Up the SAS Environment Manager Report Center

Often the first portions of the Report Center to be exploited are these three:

  • ARM Performance Reports
  • Audit Reports
  • ACM Reports

So, we’ll use these three as examples.

If you open ARM Performance Reports, you’ll find all the data about SAS artifacts, how much they are being used, and who is using them. SAS artifacts include such items as Stored Process Servers, Workspace Servers, tables, cubes, stored processes, OS directory usage, SAS procedure usage, and stored process times elapsed. They also report which users are using which types of SAS artifact. You can get Workspace Server load distribution by hour, and number of workspace server sessions broken down by day. This is a great first step for a SAS administrator to know what’s going on in his/her production deployment.

As for customizing, you’ll see this menu for all the ARM Performance Reports (and nearly all other reports as well).  You get to choose:

1.     Number of observations to print in report

2.     Subsetting, by begin/end date of data, WHERE clause

3.     Look and feel:  output format, ODS style

4.     Debugging options: show SAS log, macro calls/parameters, and macro generation (using the debug level selector)

Opening Up the SAS Environment Manager Report Center2

Careful: The WHERE clause is free-form, so you must know your variables names, types, and reasonable or valid values, or you will get errors.

The next interesting collection of reports in the Report Center is the Audit Reports, where you’ll find reports on all the events that could be scraped from SAS logs, such as:

  • Authentication errors
  • Group membership changes
  • Changes in user accounts
  • Other metadata changes
  • Client connections by userID
  • Most recent login for each userID

The most complex set of data is reported on by the ACM Reports (Agent Collected Metrics). This encompasses all the data that the SAS Environment Manager collects in its normal operations. As you would expect, a great deal of this data is the same as what you’d see when examining the metrics in SAS Environment Manager using the Resources->Browse menus.

ACM reports typically provide far more parameter options beyond the usual Titles, Footnotes, Date/Time ranges, and WHERE clause. This makes them far more customizable once you’re familiar with the additional parameters. For example, you typically have a prompt screen like the following:

Opening Up the SAS Environment Manager Report Center3

You can choose your data set, variables to display, many plotting options, X-axis and Y-axis variables and labels, BY groupings, layout and axis options, and many more. Plus, of course, the standard selectors for time/date, WHERE clauses, Titles and Footnotes.

So the lesson here is:  Know your data, know something about plotting/graphing in SAS and what the terms mean, be willing to experiment and have a little patience. With these assets you can get literally hundreds of great tables and graphs.

tags: SAS Administrators, SAS Environment Manager, SAS Professional Services

Opening Up the SAS Environment Manager Report Center was published on SAS Users.