4月 082019
 

Have you ever wondered if love at first sight really exists? And if it exists, what qualities are people drawn too? Watch any romantic comedy and you’ll see this phenomenon play out on the big screen. Which begs the question, “If it can happen to them why not me?” Let’s [...]

Love at first sight: authentic or absurd? was published on SAS Voices by Melanie Carey

4月 082019
 


As word spreads that SAS integrates with open source technologies, people are beginning to explore how to connect, interact with, and use SAS in new ways. More and more users are examining the possibilities and with this comes questions like: How do I code A, integrate B, and accomplish C?

Documentation is plentiful but is undergoing a makeover. People aren’t sure where to go for help – and that's why we're launching the SAS Developers Community, where you can gather to ask questions and get answers.

The community will mirror the activities in existing SAS Communities: Q&A, library articles, tips, technical discussions, etc. We migrated some content from other boards. For example, we moved the content from the Coding on SAS Viya board to the new community. Additionally, we scoured other boards for content that may be better aligned with developers and moved it. We also created some original content. Any good community needs participation by all, so read on and get the 411 on the new Developers Community.

Who is the target audience?

Developers – data scientists, application developers, analysts, programmers and administrators – who need to access SAS resources and/or run SAS procedures. This audience may or may not have SAS programming skills but need to access and analyze data using SAS.

What can developers expect to find?

The Developers Community provides a forum for collaboration, Q&A, and knowledge and resource sharing. The focus will be on developers using open source languages and technology. The community will create synergy between communities.sas.com, developer.sas.com, and github.com/sassoftware. SAS employees and external users will post how-to articles and other items of interest in the library section of the community. This community will not replace the SAS Programming Communities, rather, it will fill a void for non-SAS programmers who have a need/desire to interact with SAS.

When will the community launch?

The Developers Community is live! The site is public, and we've moved existing artifacts to the community. I am attending SAS Global Forum and will be available to answer questions about the new community from our booth in the Quad. Come by and see me!

Where will the community live?

The Developers Community exists on communities.sas.com, under the Developers Category.

Why do we need a community for developers?

Developers need a centralized place to share ideas, ask and answer questions, and discover resources. Currently developers lack a forum to work through things such as authentication, coding, API use, and integration issues. The community will encourage communication, engagement and leadership. Also, the Developers Community will be tightly integrated with the SAS Developers web site and SAS GitHub resources.

How do we go about creating the community?

After seeding the SAS Developer Community with existing discussions, we'll build out a group of SAS developer experts to help monitor the community. The true magic will happen as questions are asked, discussions transpire, and ideas are shared. But we need to your help too. Here is your call to action.

Share the community with your networks, buddies and even family members who may get something out of chatting it up about how to develop in SAS. The livelihood of the community hinges on user interaction. Our current and future users will thank you for it. And you may make a friend while you're at it.

Launching the Developers Community in SAS Communities was published on SAS Users.

4月 082019
 

Suppose you need to assign 100 patients equally among 3 treatment groups in a clinical study. Obviously, an equal allocation is impossible because the second number does not evenly divide the first, but you can get close by assigning 34 patients to one group and 33 to the others. Mathematically, this is a grade-school problem in integer division: simply assign floor(100/3) patients to each group, then deal with the remainders. The FLOOR function rounds numbers down.

The problem of allocating a discrete number of items to groups comes up often in computer programming. I call the solution the FLOOR-MOD trick because you can use the FLOOR function to compute the base number in each group and use the MOD function to compute the number of remainders. Although the problem is elementary, this article describes two programming tips that you can use in a vectorized computer language like SAS/IML:

  • You can compute all the group sizes in one statement. You do not need to use a loop to assign the remaining items.
  • If you enumerate the patients, you can directly assign consecutive numbers to each group. There is no need to deal with the remainders at the end of the process.

Assign items to groups, patients to treatments, or tasks to workers

Although I use patients and treatment groups for the example, I encountered this problem as part of a parallel programming problem where I wanted to assign B tasks evenly among k computational resources. In my example, there were B = 14 tasks and k = 3 resources.

Most computer languages support the FLOOR function for integer division and the MOD function for computing the remainder. The FLOOR-MOD trick works like this. If you want to divide B items into k groups, let A be the integer part of B / k and let C be the remainder, C = B - A*k. Then B = A*k + C, where C < k. In computer code, A = FLOOR(B/k) and C = MOD(B, k).

There are many ways to distribute the remaining items, but for simplicity let's give an extra item to each of the first C groups. For example, if B = 14 and k = 3, then A = floor(14/3) = 4 and C = 2. To allocate 14 items to 3 groups, give 4 items to all groups, and give an extra item to the first C=2 groups.

In a vector programming language, you can assign the items to each group without doing any looping, as shown in the following SAS/IML program:

/* Suppose you have B tasks that you want to divide as evenly as possible 
   among k Groups. How many tasks should you assign to the i_th group? */
proc iml;
/* B = total number of items or tasks (B >= 0, scalar)
   k = total number of groups or workers (k > 0, scalar)
   i = scalar or vector that specifies the group(s), max(i)<=k
   Return the number of items to allocate to the i_th group */
start AssignItems(B, k, i);
   n = floor(B / k) + (i <= mod(B, k));     /* the FLOOR-MOD trick */
   return(n);
finish;
 
/* Test the code. Assign 14 tasks to 3 Groups. */
nItems = 14;
nGroups = 3;
idx = T(1:nGroups);          /* get number of items for all groups */
Group = char(idx);           /* ID label for each group */
n = AssignItems(nItems, nGroups, idx);
print n[c={"Num Items"} r=Group L="Items per Group"];

The AssignItems function is a "one-liner." The interesting part of the AssignItems function is the binary expression i <= mod(B, k), which is valid even when i is a vector. In this example, the expression evaluates to the vector {1, 1, 0}, which assigns an extra item to each of the first two groups.

Which items are assigned to each group?

A related problem is figuring out which items get sent to which groups. In the example where B=14 and k=3, I want to put items 1-5 in the first group, items 6-10 in the second group, and items 11-14 in the last group. There is a cool programming trick, called the CUSUM-LAG trick, which enables you to find these indices. The following function is copied from my article on the CUSUM-LAG trick. After you find the number of items in each group, you can use the ByGroupIndices function to find the item numbers in each group:

/* Return kx2 matrix that contains the first and last elements for k groups that have sizes 
    s[1], s[2],...,s[k]. The i_th row contains the first and last index for the i_th group. */
start ByGroupIndices( s );
   size = colvec(s);              /* make sure you have a column vector */
   endIdx = cusum(size);          /* locations of ending index */
   beginIdx = 1 + lag(endIdx);    /* begin after each ending index ... */
   beginIdx[1] = 1;               /*    ...and at 1  */
   return ( beginIdx || endIdx );
finish;
 
/* apply the CUSUM-LAG trick to the item allocation problem */
GroupInfo = n || ByGroupIndices(n);
print GroupInfo[r=Group c={"NumItems" "FirstItem" "LastItem"}];

The table shows the number of items allocated to each group, as well as the item indices in each group. You can use the FLOOR-MOD trick to get the number of items and the CUSUM-LAG trick to get the indices. In a vector language, you can implement the entire computation without using any loops.

The post Use the FLOOR-MOD trick to allocate items to groups appeared first on The DO Loop.

4月 082019
 
The catch phrase “everything happens somewhere” is increasingly common these days.  That “somewhere” translates into a location on the Earth; a latitude and longitude.  When one of these “somewhere’s” is combined with many other “somewhere’s”, you quickly have a robust spatial data set that becomes actionable with the right analytic tools.

Opportunities for Spatial Analytics are increasing

In today’s modern world, GPS-enabled devices are ubiquitous, and their use continues to increase daily.  Cell phones, cars, fitness trackers, and cameras are all able to locate and track our position.  As a result, the location analytics market is expected to grow to over USD 16 Billion by 2021, up 17.6% from 2016 [1].

Waldo Tobler, an American-Swiss geographer and cartographer, developed his First Law of Geography based on this concept of everything happening somewhere.  He stated, “Everything is related to everything else, but near things are more related than distant things”[2].  As analytic professionals, we are accustomed to working with these correlations using scatterplots, heatmaps, or clustering models.  But what happens when we add a geographic map into the analysis?

Maps offer the ability to unlock a new level of insight into our data that traditional graphs do not offer: personal connection.  As humans, we naturally relate to our surroundings on a spatial level.   It helps build our perspective and frame of reference through which we view and navigate the world.  We feel a sense of loss when a physical landmark from our childhood – a building, tree, park, or route we used to walk to school – is destroyed or changed from the memories we have of it.  In this sense, we are connected, spatially and emotionally, to our surroundings.

We inherently understand how data relates to the world around us, at some level, just by viewing it on a map.  Whether it is a body of water or a mountain affecting a driving route or maybe a trendy area of a city causing housing prices to increase faster than the local average, a map connects us with these facts intuitively.  We come to these basic conclusions based solely on our experiences in the world and knowledge of the physical landmarks in the map.

One of the best examples of this is the 1854 Cholera outbreak in London.  Dr. John Snow was one of the first to use a map for understanding the origin of an epidemiological outbreak.  He created a map of the affected London neighborhood by plotting the location of all known Cholera deaths.  In addition to the deaths, he also plotted the location of 13 community wells that served as the public water supply.  Using this data, he was able to see a clustering of deaths around a single pump.  Armed with this information, Dr. Snow was able to convince local officials to remove the handle from the Broad Street pump.  Once removed, new cases of Cholera quickly began to diminish.  This helped prove his theory the outbreak’s origin was not air-borne as commonly believed during that time, but rather of a water-borne origin. [3]

1854 London Cholera deaths: Tabular data vs. Coordinate map [3]

Let’s look at how Dr. Snow’s map helped mitigate the outbreak and prove his theory.  The image above compares the data of the recorded deaths and community wells in tabular form to a Coordinate map.  It is obvious from the coordinate map that there is a clustering of points.  Town officials and those familiar with the neighborhood could easily get a sense of where the outbreak was concentrated.  The map told a better story by connecting their personal experience of the area to the locations of the deaths and ultimately to the wells.  Something a data table or traditional graph could not do.

Maps of London Cholera deaths with modern analytic overlays [3]

Today, with the computing power and modern analytic methods available to us, we can take the analysis even further.  The examples above show the same coordinate map with added Voronoi polygon and cluster analysis overlays.  The concentration around the Broad Street pump becomes even clearer, showing why Geographic Maps are an important tool to have in your analytic toolbox.

SAS Global Forum 2019 is being held April 28-May 1, 2019 in Dallas, Texas.  If you are planning to go to this year’s event, be sure to attend one of our presentations on the latest mapping features included in SAS Visual Analytics and BASE SAS.  While you’re there, don’t forget to stop by the SAS Mapping booth located in the QUAD to say ‘Hi!’ and let us help with your spatial data needs.  See you in Dallas!

Introduction to Esri Integration in SAS Visual Analytics

  • Monday, April 29, 4:30-5:30p, Room: Level 1, D162

There’s a Map for That! What’s New and Coming Soon in SAS Mapping Technologies

  • Tuesday April 30, 4:00-4:30p, Room: Level 1, D162

Creating Great Maps in ODS Graphics Using the SGMAP Procedure

  • Wednesday May 01, 11:30a-12:30p, Room: Level 1, D162

[1] https://www.marketsandmarkets.com/Market-Reports/location-analytics-market-177193456.html

[2] https://en.wikipedia.org/wiki/Tobler%27s_first_law_of_geography

[3] https://www1.udel.edu/johnmack/frec682/cholera/

How the 1854 Cholera outbreak showed us the importance of spatial analysis was published on SAS Users.

4月 062019
 

Recently, the North Carolina Human Trafficking Commission hosted a regional symposium to help strengthen North Carolina’s multidisciplinary response to human trafficking. One of the speakers shared an anecdote from a busy young woman with kids. She had returned home from work and was preparing for dinner; her young son wanted [...]

Countering human trafficking using text analytics and AI was published on SAS Voices by Tom Sabo

4月 062019
 

Recently, the North Carolina Human Trafficking Commission hosted a regional symposium to help strengthen North Carolina’s multidisciplinary response to human trafficking. One of the speakers shared an anecdote from a busy young woman with kids. She had returned home from work and was preparing for dinner; her young son wanted [...]

Countering human trafficking using text analytics and AI was published on SAS Voices by Tom Sabo

4月 052019
 

Recently, you may have heard about the release of the new SAS Analytics Cloud. The platform allows fast access to data-science applications in the cloud! Running on the SAS Cloud and using the latest container technology, Analytics Cloud eliminates the need to install, update, or maintain software or related infrastructure.

SAS Machine Learning on SAS Analytics Cloud is designed for SAS and open source data scientists to gain on-demand programmatic access to SAS Viya. All the algorithms provided by SAS Visual Data Mining and Machine Learning (VDMML), SAS Visual Statistics and SAS Visual Analytics are available through the offering. Developers and data scientists access SAS through a programming interface using either the SAS or Python programming languages.

A free trial for Analytics Cloud is available, and registration is simple. The trial environment allows users to manage and collaborate with others, share data, and create runtime models to analyze their data. The system is pre-loaded with sample data for learning, and allows users to upload their own data. My colleague Joe Furbee explains how to register for the trial and takes you on a tour of the system in his article, Zero to SAS in 60 Seconds- SAS Machine Learning on SAS Analytics Cloud.

Luckily, I had the privilege of being the technical writer for the documentation for SAS Analytics Cloud, and through this met two of my now close friends at SAS.

Alyssa Andrews (pictured left) and Mariah Bragg (pictured right) are both Software Developers at SAS, but worked on the UI for SAS Analytics Cloud. Mariah works in the Research and Development (R&D) division of SAS while Alyssa works in the Information Technology (IT) division. As you can see this project ended up being an interesting mix of SAS teams!

As Mariah told me the history, I learned that SAS Analytics Cloud “was a collaborative project between IT and R&D. The IT team presented the container technology idea to Dr. Goodnight but went to R&D because they wanted this idea run like an R&D project.”

As we prepared for the release of SAS Analytics Cloud to the public, I asked Mariah and Alyssa about their experience working on the UI for SAS Analytics Cloud, and about all the work that they had completed to bring this powerful platform to life!


What is SAS Analytics Cloud for you? How do you believe it will help SAS users?

Alyssa: For me, it is SAS getting to do Software as a Service. So now you can click on our SAS Software and it can magically run without having to add the complexity of shipping a technical support agent to the customers site to install a bunch of complex software.

Mariah: I agree. This will be a great opportunity for SAS to unify and have all our SAS products on cloud.

Alyssa: Now, you can trial and then pay for SAS products on the fly without having to go through any complexities.

What did you do on the project as UI Developers?

Alyssa: I was lent out to the SAS Analytics Cloud team from another team and given a tour-of-duty because I had a background in Django (a high-level Python Web design tool) which is another type of API framework you can build a UI on top of. Then I met Mariah, who came from an Angular background, and we decided to build the project on Angular. So, I would say Mariah was the lead developer and I was learning from her. She did more of the connecting to the API backend and building the store part out, and I did more of the tweaks and the overlays.

What is something you are proud of creating for SAS Analytics Cloud?

Mariah: I’m really proud to be a part of something that uses Angular. I think I was one of the first people to start using Angular at SAS and I am so excited that we have something out there that is using this new technology. I am also really proud of how our team works together, and I’m really proud of how we architectured the application. We went through multiple redesigns, but they were very manageable, and we really built and designed such that we could pull out components and modify parts without much stress.

Alyssa: That we implemented good design practices. It is a lot more work on the front-end, but it helps so much not to have just snowflake code (a term used by developers to describe code that isn’t reusable or extremely unique to where it becomes a problem later on and adds weight to the program) floating. Each piece of code is there for a reason, it’s very modular.

What are your hopes for the future of SAS Analytics Cloud?

Alyssa: I hope that it continues to grow and that we add even more applications to this new container technology, so that SAS can move even more into the cloud arena. I hope it brings success. It is a really cool platform, so I can’t wait to hear about users and their success with it.

Mariah:
I agree with Alyssa. I also hope it is successful so that we keep moving into the Cloud with SAS.

Learning more

As a Developmental Editor with SAS Press, it was a new and engaging experience to get to work with such an innovative technology like SAS Analytics Cloud. I was happy I got to work with such an exciting team and I also look forward to what is next for SAS Analytics Cloud.

And as a SAS Press team member, I hope you check out the new way to trial SAS Machine Learning with SAS Analytics Cloud. And while you are learning SAS, check out some of our great books that can help you get started with SAS Studio, like Ron Cody’s Biostatistics by Example Using SAS® Studio and also explore Geoff Der and Brian Everitt’s Essential Statistics Using SAS® University Edition.

Already experienced but want to know more about how to integrate R and Python into SAS? Check out Kevin D. Smith’s blogs on R and Python with SAS Viya. Also take a moment to investigate our new books on using open source R and Python with SAS Viya: SAS Viya: The R Perspective by Yue Qi, Kevin D. Smith, and XingXing Meng and SAS Viya: The Phyton Perspective by Kevin D. Smith and XingXing Meng.

These great books can set you on the right path to learning SAS before you begin your jump into SAS Analytics Cloud, the new way to experience SAS.

SAS® Analytics Cloud—an interview with the women involved was published on SAS Users.

4月 052019
 

You are a data scientist, in your office, doing data scientist-y things when, your manager's, manager's, manager makes an impossible request. She wants you take a raw data set from the stem cell research team, scrub the data, create and score models, and be ready to rescore when new data comes is available. And she wants it in a week. WHAT?! Your company doesn't own an analytics software license, and a spreadsheet is not going to work on this data with millions of records. Even if you received funding, how could you ever create and maintain an environment under your tight deadline? Take a deep breath, conjure your inner data scientist acumen, and realize SAS has the answer.

SAS Machine Learning on SAS Analytics Cloud provides on-demand programming access to machine learning algorithms in the cloud. No downloads, no install, no infrastructure, no maintenance. This solution provides a multithreaded, multiuser environment for concurrent access to data in memory. The solution is designed for data scientists (and others) coding in SAS or Python and allows them on-demand programmatic access to SAS Viya. You can find more details on Analytic Cloud in the fact sheet. You can even try it for free! The rest of this article will walk you through the features of this new SAS offering and outline how it can help you complete the task bestowed upon you.

Register and get started

Literally, to sign up for the trial, all you need are a SAS Profile, an email address, and a PC. You will be coding in SAS in less than a minute. From the SAS Cloud Analytics page, select the Get Free Trial button. This takes you to the SAS Profile login page (note you can create your SAS Profile here if you do not have one).

SAS Profile log in or creation

Agree to the Terms and Conditions on the License Agreement page and select the Continue button:

Trial License Agreement

You will receive an email containing a URL much like the following:

email confirmation with trial URL

Logging in

Select the link or paste it into your browser (Google Chrome 64-bit recommended) and you will see the log in screen. Enter your SAS Profile credentials and click the Sign In button.

Sign In screen

The Home screen (Applications) appears.

Home Page

We'll discuss the Data and Team pages in further detail later on in this article. You have two options for applications: SAS Studio (for SAS programming) and JupyterLab (for Python programming). This article focuses on SAS Studio. A follow up article will cover the JupyterLab use case. Select the SAS Studio button, a new tab opens to SAS Studio, and we're ready to start coding.

SAS Studio

You are familiar with the SAS language, but you need to brush up a little. Have no fear, support documentation is easily accessible. Also, the SAS Data Mining and Machine Learning Community is a great place to discover additional resources and ask questions. Finally, embedded in SAS Studio are code snippets. You decide to explore the latter.

Code snippets

In SAS Studio select the Snippets twisty in the left pane. Navigate to the SAS Viya Machine Learning section. Here you find code samples you will use to prep and analyze your data. When opening a snippet, you see code and detailed comments on what the code will accomplish. You will use these snippets as a guide when you load and prep your data and preform your analysis. Below is an image of the Prepare and Explore Data snippet. Notice each code step has accompanying comments.

You read through each snippet in the Machine Learning section. The command and structure of the code comes back to you pretty quickly and you're now ready to try it all out on your own data.

Uploading data

Now that you have an idea of what code you need to write, you need to load the data from the research department. You accomplish this by selecting the Server Files and Folders twisty and navigate to the Folder Shortcuts section. In this instance you want to upload your file into the shared/data directory (I'll explain why I chose this location in a moment). Use the Upload button to upload the research data file.

Upload file to the data directory

You're not alone

Files uploaded to shared/data are now visible by others logged into the environment. Wait, did I forget to mention this is a multi-user environment?! Well, yes, it is. You can invite others to collaborate on the project. To add and manage users, return to the Home screen (leaving SAS Studio open). Select the Team section in the left pane. The Team page lists users and displays an Invite button, used to send an invitation for system access to others.

Teams page

To invite others, click the button and enter the email address of the new user. This generates and sends an invitation in an email. The new user accepts the invite and now has access to the system. Using the URL provided in the email the new user logs in using their own SAS Profile credentials. The default role for new users is ‘User.’ A user with admin privileges can change the role to ‘Admin. In the free trial, you are permitted to have a total of five users.

Shared data

You may have guessed by now the Data section lists directories and files located in the shared directory in SAS Studio.

Data page

You also notice here you have 5 GB of storage space. This includes shared and non-shared files.

I love this. How do I get more?

Now you know your way around the system and are ready to start coding. Return to SAS Studio, open a new program, and commence your analysis of the stem cell data. When you successfully deliver the project and impress your management chain, you can mention how the SAS Analytics Cloud solution made it all possible (and simple). You now have a case for the departmental procurement of the solution opening your organization up to add more users, access more storage, and gain more power to run advanced machine learning algorithms on your data.

Your turn

In this article I've outlined how to easily register for the SAS Machine Learning trial and start coding in the matter of a minute. Try it out yourself. Register, load your data, get coding, and solve your problem.

Related Resources

For more details on the development of SAS Analytics Cloud, check out Missy Hannah's interview with two UI developers on the project.

Zero to SAS in 60 Seconds- SAS Machine Learning on SAS Analytics Cloud was published on SAS Users.

4月 052019
 

Have you ever dreamed of working for a professional sports organization? Do you play fantasy sports leagues and fantasize about owning a real team? Do you follow the news about player drafts and trades, and wish you could influence who your team picks? Well, here’s your chance. The latest SAS [...]

What’s it like to recruit the next great soccer star? Find out in this SAS Data Science Escape Room was published on SAS Voices by Janny Morsink

4月 052019
 

When working with files like SAS programs, images, documents, logs, etc., we are used to accessing them in operating system directories. In Viya, many of these files are not stored on the file-system. In this blog, we will look at where and how files are stored in Viya, and how to manage them.

In Viya, external files are stored in the Infrastructure Data server and accessed using the file micro-service. The file service manages a wide variety of file types including, images, html files, text files, csv files, SAS programs, media files, pdfs, office documents and logs. The file service is not a complete file system, but rather a method of accessing individual files stored in the infrastructure data server identified via their URI (a unique identifier).

Some files may also be accessible in Viya folders. The file service stores resources like images, SAS programs, etc. in the Infrastructure Data server, but they are accessible from Viya folders in the Content Area of Environment Manager or in SAS Drive.

The screenshot below shows the properties of an image file that was uploaded to a Viya folder using SAS Drive. You can see the filename, the URI (including the ID) and the location in the folder structure.

For files that are surfaced in folders, we can manage the files (view, copy, move, delete etc.) using SAS Environment Manager Content area, SAS Drive or the Folders command line interface. To learn more, watch this video from the SAS Technical Insights and Expertise Series.

However, many system generated files are managed by the file service and stored in the infrastructure data server, but are not accessible from the folders interfaces. A good example of system generated files that an administrator may need to manage are logs.

Many times when processing in Viya, logs are created and stored in the infrastructure data server. For example, when CAS table state management jobs, data plans, file imports, model or scoring processes are executed, a log is generated and stored by the file service.
How do we manage these files? While the files generated are mostly small in size, a large active Viya system with a long history will need management of the log and other files stored in the infrastructure data server.

One way we can access these files is using the REST API of the file service.

You can use your favorite tool to access the REST API. In this blog, I will use the GEL pyviyatools, initially the basic callrestapi tool. For more details on the pyviyatools, read this blog or go directly to the SAS GitHub site.

To get a list of all files are stored in the infrastructure data server send a GET request to the /files/files endpoint of the files service.

/./callrestapi.py -m get -e /files/files -o simple/

Partial output shows the image file referenced above and what could be a system generated log file.
=====Item 0 =======
contentType = image/jpeg
createdBy = geladm
creationTimeStamp = 2019-01-24T14:40:15.085Z
description = None
encoding = UTF-8
id = 6d2995b3-0fa6-4090-a338-1fbfe51fb26b
modifiedBy = geladm
modifiedTimeStamp = 2019-01-25T17:59:28.500Z
name = company_logo.JPG
properties = {}
size = 327848
version = 2

=====Item 164 =======
contentType = text/plain
createdBy = sasadm
creationTimeStamp = 2019-01-31T12:35:00.902Z
description = None
encoding = UTF-8
id = e4d4c83d-8677-433a-a8d3-b03bf00a5768
modifiedBy = sasadm
modifiedTimeStamp = 2019-01-31T12:35:09.342Z
name = 2019-01-31T12:35.823Z-results.log
parentUri = /jobExecution/jobs/380d3a0c-1c31-4ada-bf8d-4db66e786669
properties = {}
size = 2377
version = 2

To see the content of a file, use a GET request, the id of the file and the content endpoint. The content of the log file is displayed in the call below.

/./callrestapi.py -m get -e /files/files/01eb020f-468a-49df-a05a-34c6f834bfb6/content/

This will display the file contents. The output shows the log from a CAS table state management job.

———-JOB INFORMATION———-

Job Created: 2019-01-31T12:35.823Z
Job ID: 380d3a0c-1c31-4ada-bf8d-4db66e786669
Heartbeat interval: PT0.3S
Job expires after: PT168H
Running as: sasadm
Log file: /files/files/e4d4c83d-8677-433a-a8d3-b03bf00a5768 [ 2019-01-31T12:35.823Z-results.log ]
Arguments:

Options:
{
“enabled” : true,
“type” : “LOAD”,
“settings” : {
“refresh” : false,
“refreshMode” : “newer”,
“refreshAccessThreshold” : 0,
“varChars” : false,
“getNames” : true,
“allowTruncation” : true,
“charMultiplier” : 2,
“stripBlanks” : false,
“guessRows” : 200,
“scope” : “global”,
“encoding” : “utf-8”,
“delimiter” : “,”,
“successJobId” : “”
},
“selectors” : [ {
“serverName” : “cas-shared-default”,
“inputCaslib” : “hrdl”,
“outputCaslib” : “hrdl”,
“filter” : “or(endsWith(tableReference.sourceTableName,’.sashdat’), endsWith(tableReference.sourceTableName,’.SASHDAT’),\nendsWith(tableReference.sourceTableName,’.sas7bdat’),\nendsWith(tableReference.sourceTableName,’.csv’)\n)”,
“settings” : { }
} ]
}

————————————————–

Created session cas-shared-default: d4d1235a-6219-4649-940c-e67526be31ed (CAS Manager:Thu Jan 31 07:35:01 2019)
Using session d4d1235a-6219-4649-940c-e67526be31ed
Server: cas-shared-default
Input caslib: hrdl
Output caslib: hrdl
Effective Settings:

———-LOAD STARTING———-

— loaded –> unloaded – HR_SUMMARY unloaded – HR_SUMMARY_NEW unloaded – HRDATA unloaded – PERFORMANCE_LOOKUP <– performance_lookup.sas7bdat
Access denied.
———-LOAD COMPLETE———-

———-CLEANUP STARTING———-

Session deleted: cas-shared-default: d4d1235a-6219-4649-940c-e67526be31ed
———-CLEANUP COMPLETE———-

Final Job State: completed
Log file:/files/files/e4d4c83d-8677-433a-a8d3-b03bf00a5768

I started my journey into the file service to discover how an administrator could manage the log files. In reviewing the files and their contents using the REST API, I discovered that there was no easy way to uniquely identify a log file. The table below shows the attributes of some of log files from the GEL Shared Viya environment.

As this post has illustrated, file service REST API can be used to list and view the files. It can also be used for other file management activities. To help administrators manage files that are not visible via a user interface, a couple of new tools have been added to the pyiyatools.

listfiles.py provides an easy interface to query what files are currently stored in the infrastructure data server. You can sort files by size or modified date, and query based on date modified, user who last modified the file, parentUri or filename. The output provides the size of each file so that you can check the space being used to store files. Use this tool to view files managed by the file service and stored in the infrastructure data server. You can use lisfiles.py -h to see the parameters.

For example, if I want to see all potential log files older than 6 days old created by the /jobexecution service, I would use:

/./listfiles.py -n log -p /jobExecution -d 6 -o csv/

The output is a list of files in csv format:

id ,name ,contentType ,documentType ,createdBy ,modifiedTimeStamp ,size ,parentUri

“f9b11468-4417-4944-8619-e9ea9cd3fab8″,”2019-01-25T13:35.904Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T13:35:09.490Z”,”2459″,”/jobExecution/jobs/37536453-fe2f-41c2-ba63-ce72737e482c”

“dffdcc97-3fb0-47c1-a47d-e6c8f24f8077″,”2019-01-25T12:35.869Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T12:35:08.739Z”,”2459″,”/jobExecution/jobs/99708443-2449-40b7-acc3-c313c5dbca23″

“5fa889b6-93fb-4496-98ba-e0c055ca5999″,”2019-01-25T11:35.675Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T11:35:09.118Z”,”2459″,”/jobExecution/jobs/eb182f88-4853-41f4-be24-225176991e8a”

“87988659-3c2d-4602-b61a-8042b34022ac”,”2019-01-25T10:35.657Z-results.log”,”text/plain”,”None”,”sasadm”,”2019-01-25T10:35:09.881Z”,”2459″,”/jobExecution/jobs/73fffe47-a7ef-4c1d-b7bf-83ef0b86319e”

archivefiles.py allows you to read files from the file service and save them to a directory on the file system. Optionally, the tool will also delete files from the file service to free up space. For example, if I want to archive all the files I listed above, I would use:

/./archivefiles.py -n log -d 6 -p /job -fp /tmp/

This tool will create a timestamp directory under /tmp and save a copy of each file in the directory.

If you want to archive and delete, add the -x option.

IMPORTANT: Use the archive tool carefully. We recommend that you run a Viya Backup prior to running the tool to delete files.

Now you know where your files are and you have some help with managing them. For the full details of how you can manage files using the file service REST API you can view the file service REST API documentation on developer.sas.com. If you would like to suggest any changes to the existing tools, please enter a suggestion on GitHub.

Where are my Viya files? was published on SAS Users.