Copy Files

11月 142016

I've supplied dozens of custom tasks for SAS Enterprise Guide, but the Copy Files task is easily the most popular. The Copy Files task allows you to capture "file transfer" steps inside your process flow, so that you can automate any file upload and download operations between your PC and your SAS workspace session. It has proven to be an essential task for customers who move from using PC SAS to SAS Enterprise Guide. Many of you still need a method to copy data and results to and from your SAS session. When the SAS session is on a remote server, then this task fills that important gap.

Because "Copy files" is a custom task, you have to download the task package (from this blog) and follow a few steps to install the task into your SAS Enterprise Guide environment. When installed, the task can be found in the Tools → Add-In menu.

Copy Files task moves to the Tasks → Data menu

Copy Files in new menuThat's about to change with the next release: SAS Enterprise Guide v7.13. We're going to make an honest task out of "Copy Files," as it becomes an official feature in SAS Enterprise Guide. That's great news for a couple of reasons: no more custom install steps, and you can now get official support from SAS Tech Support when using it (although they would have always helped before now). The task works exactly the same way and if you have existing projects that use it, you don't need to make any changes. However, there is one change you need to know about: as an "official" task, it will appear in an official menu location. As of SAS Enterprise Guide 7.13, you'll find Copy Files in the Tasks → Data menu, near the bottom with some other utility-type tasks. And if you had previously installed it as a custom task, it will no longer appear in the Tools → Add-In menu.

SAS Enterprise Guide 7.13 is set to release within the next couple of weeks (near the end of November 2016), and it contains several exciting new features that I'll describe in this blog. Many of you will see it immediately when SAS Enterprise Guide prompts you to update. Stay tuned!

tags: Copy Files, SAS custom tasks, SAS Enterprise Guide

The post The Copy Files task is going legit (and moving) appeared first on The SAS Dummy.

5月 112015

I've written about how to use the FILENAME ZIP method to read and update ZIP files in your SAS programs. The ZIP method was added in SAS 9.4, and its advantage is that you can accomplish more in SAS without having to launch external utilities such as WinZip, gunzip, or 7-Zip.

Several readers replied with questions about how you can use the content of these ZIP files within your SAS program. The basic scenario is: "I've got some data files in my ZIP archive. I want to use SAS to unzip these and then use them as data within my SAS process. Can I do this?"

Yes, you can -- but it does require an extra step. Even though FILENAME ZIP can show you the contents and structure of your ZIP file, most SAS procedures cannot access the content directly while it's in the archive. So, the additional step is to copy the file to another location, effectively extracting it from the ZIP file.

As an example, I created a ZIP file with two files and a subfolder:
  |__ sas_tech_talks_15.xlsx
  |__ sas/
      |__ instanttitles.sas7bdat

This SAS program helps me to discover how FILENAME ZIP sees the file:

filename inzip ZIP "";
/* Read the "members" (files) from the ZIP file */
data contents(keep=memname isFolder);
 length memname $200 isFolder 8;
 if fid=0 then
 do i=1 to memcount;
  /* check for trailing / in folder name */
  isFolder = (first(reverse(trim(memname)))='/');
/* create a report of the ZIP contents */
title "Files in the ZIP file";
proc print data=contents noobs N;


        Files in the ZIP file                                         
 memname                       isFolder
 sas/                             1  
 sas/instanttitles.sas7bdat       0  
 sas_tech_talks_15.xlsx           0  
                N = 3

With this information, I can now "copy" the XLSX file out of the ZIP file and then import it into a SAS data set. Notice how I can use the "member" syntax (fileref with the file I want in parentheses) to address a specific file in the ZIP archive. I want to copy just from the actual files, and not the folder-level entries.

/* identify a temp folder in the WORK directory */
filename xl "%sysfunc(getoption(work))/sas_tech_talks_15.xlsx" ;
/* hat tip: "data _null_" on SAS-L */
data _null_;
   /* using member syntax here */
   infile inzip(sas_tech_talks_15.xlsx) 
       lrecl=256 recfm=F length=length eof=eof unbuf;
   file   xl lrecl=256 recfm=N;
   put _infile_ $varying256. length;
proc import datafile=xl dbms=xlsx out=confirmed replace;

Sample output from my SAS log:

NOTE: The infile INZIP(sas_tech_talks_15.xlsx) is:,
      Member Name=sas_tech_talks_15.xlsx

NOTE: UNBUFFERED is the default with RECFM=N.
NOTE: The file XL is:
      Filename=C:SAS Temporary Files_TD396_Prc2sas_tech_talks_15.xlsx,
      RECFM=N,LRECL=256,File Size (bytes)=0,
      Last Modified=11May2015:11:38:59,
      Create Time=11May2015:11:20:23

NOTE: A total of 55 records were read from the infile library INZIP.
NOTE: 55 records were read from the infile INZIP(sas_tech_talks_15.xlsx).
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds

To use the SAS data set in the file, I need to copy it into a location shared by a SAS library. In this example, I will again use the WORK location. Because my SAS data set is in a logical subfolder (named "sas") within the archive, I need to include that path as part of the member syntax on the INFILE statement.

/* Copy a zipped data set into the WORK library */
filename ds "%sysfunc(getoption(work))/instanttitles.sas7bdat" ;
data _null_;
   /* reference the member name WITH folder path */
   infile inzip(sas/instanttitles.sas7bdat) 
	  lrecl=256 recfm=F length=length eof=eof unbuf;
   file   ds lrecl=256 recfm=N;
   put _infile_ $varying256. length;
proc contents data=work.instanttitles;

Partial output in my example:

                             Files in the ZIP file                          
                             The CONTENTS Procedure

 Data Set Name        WORK.INSTANTTITLES            Observations          1475
 Member Type          DATA                          Variables             6   
 Engine               V9                            Indexes               0   
 Created              01/29/2015 15:09:54           Observation Length    248 
 Last Modified        01/29/2015 15:09:54           Deleted Observations  0   
 Protection                                         Compressed            NO  
 Data Set Type                                      Sorted                NO  
 Data Representation  WINDOWS_64                                              
 Encoding             wlatin1  Western (Windows)                              

Of course, all of this can be automated even further by writing SAS code that automatically iterates through the ZIP file member names and copies/imports each of the members as needed.

tags: Copy Files, FILENAME ZIP, SAS 9.4, ZIP files

The post Using FILENAME ZIP to unzip and read data files in SAS appeared first on The SAS Dummy.

3月 122015

Two popular SAS custom tasks have recently been updated for SAS Enterprise Guide 7.1. Most custom tasks that I've shared will work without modification across releases, but these two required a special rebuild due to some internal product API changes.

The Project Reviewer task allows you to see a detail view of the tasks/programs within your SAS Enterprise Guide process flows. The listing shows the task names and types, when they were added to the project and last modified, and how much time they take to run. You can also create a SAS-based ODS report from the data, which is a great way to use SAS Enterprise Guide to report on your SAS Enterprise Guide project. (Wow, that's so meta!)

Download Project Reviewer: See this blog post for more information and download link

The Copy Files task is used to move files from your local PC to your SAS session, or from your SAS session to your local PC. This allows you to capture these file transfer steps within your process flow so that they are repeatable. The task supports the use of macro variables and wildcards, which gives you lots of flexibility when designing a process in which the file names and quantities are subject to change.

Copy Files task
Download Copy Files: See this blog post for more information and download link

If you use these tasks (or other custom tasks) and find them useful, please leave a comment! I love to hear how the tasks are being used. And sometimes useful custom tasks find their way into the actual product in a future release. Your feedback is important to help to make that happen.

Other related articles

tags: Copy Files, process flows, SAS custom tasks, SAS Enterprise Guide
9月 182013

A couple of years ago I shared a method for copying any file within a SAS program. It was a simple approach, copying the file byte-by-byte from one fileref (SAS file reference) to another.

My colleague Bruno Müller, a SAS trainer in Switzerland, has since provided a much more robust method. Bruno's method has several advantages:

  • It's coded as a SAS macro, so it is simple to reuse -- similar to a function.
  • It copies the file content in chunks rather than byte-by-byte, so it's more efficient.
  • It provides good error checks and reports any errors and useful diagnostics to the SAS log.
  • It's an excellent example of a well-documented SAS program!

Bruno tells me that "copying files" within a SAS program -- especially from nontraditional file systems such as Web sites -- is a common need among his SAS students. I asked Bruno for his permission to share his solution here, and he agreed.

To use the macro, you simply define two filerefs: _bcin (source) and _bcout (target), then call the %binaryFileCopy() macro. Here is an example use that copies a file from my Dropbox account:

filename _bcin TEMP;
filename _bcout "C:\temp\streaming.sas7bdat";
proc http method="get" 
%put NOTE: _bcrc=&_bcrc;
filename _bcin clear;
filename _bcout clear;

The following is partial log output from the program:

NOTE: BINARYFILECOPY start  17SEP2013:20:50:33
NOTE: BINARYFILECOPY infile=_bcin C:\SASTempFiles\_TD5888\#LN00066
NOTE: BINARYFILECOPY outfile=_bcout C:\temp\streaming.sas7bdat

NOTE: BINARYFILECOPY processed 525312 bytes
NOTE: DATA statement used (Total process time):
      real time           0.20 seconds
      cpu time            0.07 seconds    

NOTE: BINARYFILECOPY end  17SEP2013:20:50:34
NOTE: BINARYFILECOPY processtime 00:00:00.344

You can download the program -- which should work with SAS 9.2 and later -- from here:

Update: using FCOPY in SAS 9.4

Updated: 18Sep2013
Within hours of my posting here, Vince DelGobbo reminded me about the new FCOPY function SAS 9.4. With two filerefs assigned to binary-formatted files, you can use FCOPY to copy the content from one to the other. When I first tried it with my examples, I had problems because of the way FCOPY treats logical record lengths. However, Jason Secosky (the developer for FCOPY and tons of other SAS functions) told me that if I use RECFM=N on each FILENAME statement, the LRECL would not be a problem. And of course, he was correct.

Here's my example revisited:

filename _bcin TEMP recfm=n /* RECFM=N needed for a binary copy */;
filename _bcout "C:\temp\streaming.sas7bdat" recfm=n;
proc http method="get" 
data _null_;
   length msg $ 384;
   rc=fcopy('_bcin', '_bcout');
   if rc=0 then
      put 'Copied _bcin to _bcout.';
   else do;
      put rc= msg=;
filename _bcin clear;
filename _bcout clear;
tags: Copy Files, FCOPY, macro programming, SAS 9.4, SAS programming
12月 072012

One of the problems that trips up experienced SAS users when they begin to use SAS Enterprise Guide is a result of simple geography.

The SAS Enterprise Guide application runs here, on your desktop. The SAS Workspace session (which accesses data and cranks through your analysis) runs over there, on a remote machine. If you're accustomed to "PC SAS" running all on your local box, you might be forced to rethink a few of your processes. For example, do you have Excel files on your PC to import using PROC IMPORT? Does your SAS program create output that you have to save on your PC? How will you get these files to where they need to be?

There are some strategies that help, including mapped drives, UNC paths, and network shares via a mechanism like Samba. These mechanisms allow your local SAS Enterprise Guide and your remote SAS to "see" a shared location on the network. Also, SAS Enterprise Guide has a few focused tasks that can "move" files for you under the guise of "import" and "export" steps.

But there isn't a general method to copy any file you want from your PC to the SAS Workspace, or from the SAS Workspace to your PC. Until now.

I give you: the Copy Files task for SAS Enterprise Guide.

What "Copy Files" does

The Copy Files task allows you to capture file transfer steps within your SAS Enterprise Guide process flow. You can upload files from your local PC to the remote machine where your SAS Workspace is running. And you can download files from your remote SAS Workspace to a folder on your local PC. It is similar in concept to an FTP process, except that this transfer operation uses your already-existing connection to a SAS Workspace and does not require a separate application.

Some readers might be familiar with SAS/CONNECT Data Transfer Services (PROC UPLOAD and PROC DOWNLOAD), which allows you to transfer files between two SAS sessions. Again, this is similar in concept, but since there is only one SAS session in play here, we need to use a different mechanism.

How to copy files in bulk

The Copy Files task supports two features that allow you to copy multiple files with a single step. First, you can use wildcard characters to match on multiple file names. An asterisk (*) matches all characters, in any number, before the next non-wildcard character. A question mark (?) matches any single character that occupies that position in the file name.

Second, you can use SAS macro variables and expressions to specify any part of the source file names or destination folder. Use a SAS program to determine the files you need to copy earlier in the process, assign that value to a macro variable, and then have the task "dynamically" select the correct files when it's time to copy them.

Support for Task Templates

The Copy Files task also supports a standard SAS Enterprise Guide feature: Task Templates. You can create an instance of the task that is useful in one process, then save those task settings to your personalized set of Task Templates that you can use in another process or even in another project. To save your settings as a template, right-click on the Copy Files task within your process flow and select Create Task Template. To use the new template, select it as a menu item from Tasks->Task Templates.

Limitations to note

This task does not generate a SAS program that you can reuse in a batch SAS process or SAS stored process. The task works by using SAS Workspace APIs to transfer data across your network connection, so all of the work happens "behind the scenes". However, the task does create a detailed log output that shows what files were copied, how many bytes were transferred, how long it took, and whether there were any errors.

How to get the task

You can download the Copy Files task from this link (Zip file). The download package includes three versions that depend on your version of SAS Enterprise Guide: one for v4.3, one for v5.1, and one for v4.1 (with a reduced feature set). There is also a documentation file (PDF) that provides instruction for how to install and use the task.

tags: Copy Files, FTP, PROC DOWNLOAD, PROC UPLOAD, SAS custom tasks, SAS Enterprise Guide