4月 122018

The financial services industry has witnessed considerable hype around artificial intelligence (AI) in recent months. We’re all seeing a slew of articles in the media, at conference keynote presentations and think-tanks tasked with leading the revolution. AI indeed appears to be the new gold rush for large organisations and FinTech [...]

AI for fraud detection: beyond the hype was published on SAS Voices by Sundeep Tengur

4月 112018

Sometimes it is important to ensure that a matrix has unique rows. When the data are all numeric, there is an easy way to detect (and delete!) duplicate rows in a matrix.

The main idea is to subtract one row from another. Start with the first row and subtract it from every row beneath it. If any row of the difference matrix is identically zero, then you have found a row that is identical to the first row. Then do the same thing for the second row: subtract it from the third and higher rows and see if you obtain a row of zeros. Repeat this process for the third and higher rows.

The following SAS/IML program implements this algorithm for an arbitrary numeric matrix. It returns a binary indicator variable (a column vector). The i_th row of the returned vector is 1 for rows that are duplicates of some previous row:

proc iml;
/* return binary vector that indicates which rows in a 
   numerical matrix are duplicates */
start DupRows(A);
   N = nrow(A);
   dupRows = j(N, 1, 0);         /* binary indicator matrix */
   do i = 1 to N-1;
      if dupRows[i] = 0 then do; /* skip rows that are known to be duplicates */
         r = i+1:N;              /* remaining rows */ 
         M = A[r, ]-A[i, ];      /* subtract current row from remaining rows */
         b = M[ ,##];            /* sum of squares = 0 iff duplicate row */
         idx = loc( b=0 );       /* any duplicate rows for current row? */
         if ncol(idx) > 0 then
            dupRows[r[idx]] = 1; /* if so, flag them */
   return dupRows;

To test the function, consider the following 6 x 3 matrix. You can see by inspection that the matrix has three duplicate rows: the third, fifth, and sixth rows. You can call the DupRows function and print the matrix adjacent to the binary vector that indicates the duplicate rows:

A = {1 1 1,
     1 2 3,
     1 1 1,  /* duplicate row */
     3 2 1,
     3 2 1,  /* duplicate row */
     1 1 1};  /* duplicate row */
DupRows = DupRows(A);
print A DupRows;

You can use the DupRows function to write a function that excludes the duplicate rows in a matrix and returns only the unique rows, as follows:

/* return the unique rows in a numerical matrix */
start UniqueRows(A);
   uniqRows = loc(DupRows(A)=0);
   return(A[uniqRows, ]);   /* return rows that are NOT duplicates */
U = UniqueRows(A);
print U;

I like this algorithm because it uses subtraction, which is a very fast operation. However, This algorithm has a complexity of O(n(n-1)/2), where n is the number of rows in the matrix, so it is best for small to moderate values of n.

For long and skinny matrices (or for character data), it might be more efficient to sort the matrix as I showed in a previous article about how to find (and count) the unique rows of a matrix. However, the sorting algorithm requires that you sort the data by ALL columns, which can be inefficient for very wide and short data tables. For small and medium-sized data, you can use either method to find the unique rows in a matrix.

This article discusses how to remove duplicates records for numerical matrices in SAS/IML. In Base SAS, you can use PROC SORT or PROC SQL to remove duplicate records from a SAS DATA set, as shown in Lafler (2017) and other references.

The post Find the unique rows of a numeric matrix appeared first on The DO Loop.

4月 102018

SAS has never been able to sit idle and watch from the sidelines as crises ensue. Why? Because we know that good can be made possible if data is put to work. SAS Global Forum opening session spotlighted four real-life examples of how SAS is showing up and improving the [...]

What can be achieved when passion, curiosity and analytics come together? was published on SAS Voices by Anjelica Cummings

4月 092018

Bright lights, a packed house, loud music and high energy – these are the elements that come to life every year in the  SAS Global Forum opening session. But it’s not just about the entertainment and energy. Each year, the Sunday night opening session sets the stage for conference attendees. [...]

Ask the question, and watch new discoveries unfold was published on SAS Voices by Anjelica Cummings

4月 092018

When we breathe, we breathe in and breathe out. If we choose only one or the other, the results are disastrous.

The same principle applies to professional growth and development. Whether we are programmers, statisticians, teachers, students, or writers, we benefit from taking in and giving back. We "take in" when we learn something new: a new fact, a new skill, or a new technique. We "give back" when we share one of our talents with someone else, such as when we teach, mentor, or coach. Taking in and giving back reinforce each other and lead to a virtuous cycle of learning and sharing.

The COO at SAS, Oliver Schabenberger, often speaks about "lifelong learning" and encourages employees to grow and learn. Learning prevents stagnation, fosters creativity, and benefits the company as well as the individual. SAS also encourages giving back by promoting the Data For Good movement, STEM education, community outreach, and more.

In my own career, writing this blog promotes my professional growth, as does my involvement in the SAS Support Communities. These activities require me to take in and give back. I take in when I research a blog topic, when I read a journal article, and when I search the SAS documentation. I give back when I share a programming trick, a statistical method, or an interesting mathematical tidbit.

Conferences provide a unique opportunity for taking in and giving back. This week I am at SAS Global Forum, where I will give several presentations and talk with SAS customers about their data and business problems. But I will also take in information from the other attendees. I expect to learn as much as I teach. I expect to return home full of energy and excitement from days of give and take.

Breathing in and breathing out is easy. Taking in and giving back can be harder when you face the day-to-day demands of a job and family. How do you make time in your day to learn new things? How do you share your knowledge? Are these activities valued at your company? Leave a comment.

The post Taking in. Giving back. appeared first on The DO Loop.

4月 072018

I’m excited because in a couple days I will fly to Denver for SAS Global Forum 2018, the biggest SAS conference of the year, attended by thousands.

If you are coming, I hope you will attend the presentation that Lora Delwiche and I have.  It is “SAS Studio: A New Way to Program in SAS” on Wednesday April 11, 10:00-11:30am in Meeting Room506.

It’s a Hands On Workshop so it’s best if you can be there in person.  If you can’t, you can still read the paper.

In fact, you can access the proceedings for the entire conference.

And better still, you can attend the conference online.  Selected events can be viewed live including the Opening Session Sunday, April 8, 7:00-8:30pm (MT).

And if you miss that, there is always a library of interesting videos available on demand.


4月 072018

Recently I was honored to be interviewed by Mamadou Dakouo for his company DataSteps.  I was asked three questions:

  • How did I learn SAS?
  • What advice do I have for someone who wants to become a “top SAS programmer?”
  • What is my favorite SAS procedure?

My answers were

  • In graduate school
  • Be a self-starter
  • PROC FORMAT (runner-up: PROC SGPLOT)

For the details you can watch the video.


4月 062018

SAS Global Forum 2018 is just days away, and if you're one of the 6,000 SAS users heading to Denver for the event, get ready for four days of learning from SAS peers, exchanging ideas, discovering new techniques for using SAS, and a bit of fun as well.

But what if you can't make the trip to Denver this year? Is there another way to experience some of the great content that will be shared there? I'm happy to say the answer is yes!

SAS Global Forum 2018 Virtual Event

SAS Global Forum 2018 Virtual EventOnce again, SAS will provide dozens of hours of live video streaming from the event. Register for an All-Access Pass and watch select sessions including Opening Session, keynote talks, select breakouts, Tech Talks, updates from The Quad, interviews with SAS executives and developers, and more. You can find livestream presentations and add them to your calendar here. Coverage will be available for on-demand viewing on SAS Global Forum Video Portal after the conference as well.

Get involved via Social Media

Video not your thing? No worries. SAS will provide several other ways to stay up to date. For starters, you can read any of a number of blog posts from the event. Posts will come from many different SAS blogs, but all posts from SAS Global Forum will be aggregated here.

If you're on LinkedIn, Twitter or Facebook, you can stay connected with what's happening and engage with attendees on SAS’ social media channels. Join the conversation, comment on some of the cool presentations you attended or viewed, discuss the exciting news coming out of the event, or simply follow along. The channels sure to have the most activity are the SAS Users LinkedIn group, the SAS Twitter account, the SAS Users Group Facebook page; and Instagram. The hashtag for SAS Global Forum is #SASGF; be sure to use the hashtag in all your posts.

SAS Global Forum 2018 Proceedings

If you want a deeper dive into a particular talk, or if you're interested in presentations that may not be part of the Live Stream, the proceedings for SAS Global Forum 2018 are already available. Here you'll find abstracts, PDF of papers, e-posters, slides, data files, and more.

With all the opportunities to follow along, connect and contribute, you can be a part of SAS Global Forum 2018, whether you're attending in person or not.

Enjoy the event...whether you're going or not!

Participate in SAS Global Forum 2018...even if you're not going was published on SAS Users.