5月 092017
 

What do you get when you combine analytics, aviation and the Internet of Things? A learning experience that leaves everyone flying high! At Data on the Fly, 25 area high school students had the opportunity to learn how technology has changed – and continues to change – the aviation industry. [...]

Students visualize aircraft data in real time using SAS was published on SAS Voices by Katie Howard

5月 092017
 

Microservices are a key component of the SAS Viya architecture. In this post, I’ll introduce and explain the benefits of microservices. In a future post we’ll dig deeper into the microservices architecture.

What are microservices?

When we look at SAS Viya architecture diagrams, we can find, among the new core components, microservices.

Microservices are self-contained, lightweight pieces of software that

  • Do one thing.
  • Depend on one another to the least extent possible.
  • Are deployed independently.
  • Provide a language-agnostic API.
  • Can run one or more instances of these processes at any given time.

Note that the prefix “micro” doesn’t mean small in CPU or memory consumption. Rather, it refers to the software performing a single function or being narrow in scope.

Let’s compare to SAS 9

The SAS 9 Web Infrastructure Platform services and the overall platform are tightly coupled to metadata structures and schemas. Every maintenance action takes a bit of effort: can you apply a fix to a single application without first stopping the whole infrastructure? Can you upgrade one component and leave all of the other ones at the previous release? Can you…?

To address these and other issues, SAS R&D decomposed the metadata server,  the Web Infrastructure Platform, and  many web applications. As a result, we got many functional units. Each one is a microservice.

Let’s have a look at the following examples.

In SAS 9.4 we can open the SAS Management Console to manage users and groups:

In SAS Viya, we can do the same using the SAS Environment Manager web application:

You may think we simply switched to a different, web-based client. Actually, the real difference lies in the backend implementation. With SAS 9, the metadata server was responsible for servicing that functionality in addition to a host of other features. With SAS Viya, we have a dedicated microservice for it: the Identities microservice.

Here’s another example. We want to edit an option in the configuration of an application, like the address of the Open Street Map server to use with Visual Analytics geo maps. With SAS 9, we use the SAS Management Console to interact, as usual, with the metadata server.

With SAS Viya, we set the property with Environment Manager. And, guess what? We are using the Configuration microservice.

If you are curious and want to see a list of all the microservices deployed in your SAS Viya environment, you can, again, use the Environment Manager.

Note that in all these examples, the Environment Manager is simply serving as the GUI to a particular microservice supporting the associated feature.

What are the benefits of Microservices?

The move to a microservice-oriented architecture brings many benefits to all stakeholders, first and foremost to SAS users and SAS administrators.

Microservices are independently updatable

It is now easier for you to manage and maintain your environment. Hot fixes for a specific microservice are released just as normal updates, and the official installation process is documented in the

Just as with the previous point, there are a few exceptions: almost everything requires the SASLogon and Identities microservices, so, if they are down, nothing works.

Scalability and High Availability

When microservices are spun up, they self-register, making themselves available for processing requests. This way, supporting failover is as easy as ensuring you have at least two instances of the associated microservice up and running. It is possible to scale further for increased capacity/performance, and you can do so at the microservice level, based on the specific demand for each function (e.g., you likely won’t need as many instances of the Import VA SPK microservice as you do for the Authorization microservice).

Microservices are “open”

Microservices can run in different environments – bare OS, Cloud Foundry, Docker. Also, they are accessible to non-SAS developers through REST APIs. As an example, let’s say I’d like to retrieve the same properties for the SAS Administrators group that were shown above in Environment Manager. It’s as easy as calling a REST endpoint: http://<myserver>/identities/groups/SASAdministrators
The result can be in either XML or json.

In fact, even microservices communicate with one another using REST interfaces!

I hope this blog has been helpful.

Feel free to add comments or questions below.

Let’s talk about Microservices was published on SAS Users.

5月 092017
 

Microservices are a key component of the SAS Viya architecture. In this post, I’ll introduce and explain the benefits of microservices. In a future post we’ll dig deeper into the microservices architecture.

What are microservices?

When we look at SAS Viya architecture diagrams, we can find, among the new core components, microservices.

Microservices are self-contained, lightweight pieces of software that

  • Do one thing.
  • Depend on one another to the least extent possible.
  • Are deployed independently.
  • Provide a language-agnostic API.
  • Can run one or more instances of these processes at any given time.

Note that the prefix “micro” doesn’t mean small in CPU or memory consumption. Rather, it refers to the software performing a single function or being narrow in scope.

Let’s compare to SAS 9

The SAS 9 Web Infrastructure Platform services and the overall platform are tightly coupled to metadata structures and schemas. Every maintenance action takes a bit of effort: can you apply a fix to a single application without first stopping the whole infrastructure? Can you upgrade one component and leave all of the other ones at the previous release? Can you…?

To address these and other issues, SAS R&D decomposed the metadata server,  the Web Infrastructure Platform, and  many web applications. As a result, we got many functional units. Each one is a microservice.

Let’s have a look at the following examples.

In SAS 9.4 we can open the SAS Management Console to manage users and groups:

In SAS Viya, we can do the same using the SAS Environment Manager web application:

You may think we simply switched to a different, web-based client. Actually, the real difference lies in the backend implementation. With SAS 9, the metadata server was responsible for servicing that functionality in addition to a host of other features. With SAS Viya, we have a dedicated microservice for it: the Identities microservice.

Here’s another example. We want to edit an option in the configuration of an application, like the address of the Open Street Map server to use with Visual Analytics geo maps. With SAS 9, we use the SAS Management Console to interact, as usual, with the metadata server.

With SAS Viya, we set the property with Environment Manager. And, guess what? We are using the Configuration microservice.

If you are curious and want to see a list of all the microservices deployed in your SAS Viya environment, you can, again, use the Environment Manager.

Note that in all these examples, the Environment Manager is simply serving as the GUI to a particular microservice supporting the associated feature.

What are the benefits of Microservices?

The move to a microservice-oriented architecture brings many benefits to all stakeholders, first and foremost to SAS users and SAS administrators.

Microservices are independently updatable

It is now easier for you to manage and maintain your environment. Hot fixes for a specific microservice are released just as normal updates, and the official installation process is documented in the

Just as with the previous point, there are a few exceptions: almost everything requires the SASLogon and Identities microservices, so, if they are down, nothing works.

Scalability and High Availability

When microservices are spun up, they self-register, making themselves available for processing requests. This way, supporting failover is as easy as ensuring you have at least two instances of the associated microservice up and running. It is possible to scale further for increased capacity/performance, and you can do so at the microservice level, based on the specific demand for each function (e.g., you likely won’t need as many instances of the Import VA SPK microservice as you do for the Authorization microservice).

Microservices are “open”

Microservices can run in different environments – bare OS, Cloud Foundry, Docker. Also, they are accessible to non-SAS developers through REST APIs. As an example, let’s say I’d like to retrieve the same properties for the SAS Administrators group that were shown above in Environment Manager. It’s as easy as calling a REST endpoint: http://<myserver>/identities/groups/SASAdministrators
The result can be in either XML or json.

In fact, even microservices communicate with one another using REST interfaces!

I hope this blog has been helpful.

Feel free to add comments or questions below.

Let’s talk about Microservices was published on SAS Users.

5月 082017
 

Quick! What is the next term in the numerical sequence 1, 2, 1, 2, 3, 4, 5, 4, 3, 4, ...? If you said '3', then you must be an American history expert, because that sequence represents the number of living US presidents beginning with Washington's inauguration on 30APR1789 and continuing until the death of James Monroe on 04JUL1831.

I stumbled upon the Wikipedia page for "Living Presidents of the United State," which contains a table that shows the dates of inaugurations and deaths of US presidents and the number of living presidents between events. The table is very crowded and I found it difficult to visualize the data, so I decided to create a SAS graph that shows the timeline for the number of living US presidents.

All the information in the complex Wikipedia table can be derived by knowing the dates on which a new president was inaugurated or a previous president died. You can create a simple data set with that information and use SAS to calculate other information, such as the number of living presidents or the time span between events. You can download the data and the SAS program that creates the following graph. (Click to enlarge.) If you create the graph in SAS, you can hover the mouse over a marker to see which president died or was inaugurated for each event.

Timeline of living US presidents

A few interesting facts that are revealed by the visualization of these events (all statistics as of 08MAY2017):

  • The time-weighted average since Washington's inauguration is 3.43 living presidents per day.
  • The period from 1989 to 2017 featured a larger-than-usual number of living presidents. Unfortunately, I don't expect that trend to last, since Presidents Carter and G. H. W. Bush are both very old.
  • There have been six brief intervals when the only living president was the sitting president.
  • On 04JUL1826, the US lost two former presidents (John Adams and Thomas Jefferson) within 6 hours. You can see the graph drop by 2 in 1826. Interestingly, James Monroe also died on Independence Day!
  • When a president dies in office, the number of living presidents does not change, since the vice-president is inaugurated that same day. Can you spot the eight times that a president died in office?
  • No presidents died during F. D. Roosevelt's administration—except FDR himself. The 20-year period from 1933-1953 is the longest span during which the number of living presidents stayed constant.
  • The reason no one died during FDR's terms was that Herbert Hoover remained alive. Hoover had the record of being the ex-president who lived longest after his inauguration: from 1929 until 1964, which is 13,014 days or 35.65 years! However, Jimmy Carter broke that record a few years ago. Carter was inaugurated in 1977 and has lived more than 40 years since that event.

Do you notice any other interesting features of this timeline? Leave a comment.

The post Timeline of living US presidents appeared first on The DO Loop.

5月 082017
 

In his recent article Perceptions of probability, Rick Wicklin explores how vague statements about "likeliness" translate into probabilities that we can express numerically. It's a fun, informative post -- I recommend it! You'll "Almost Certainly" enjoy it.

To prepare the article, Rick first had to download the source data from the study he cited. The data was shared as a CSV file on GitHub. Rick also had to rename the variables (column names) from the data table so that they are easier to code within SAS. Traditionally, SAS variable names must adhere to a few common programming rules: they must be alphanumeric, begin with a letter, and contain no spaces or special characters. The complete rules are documented in the this method for reading data from a cloud service like DropBox and GitHub. It's still my favorite technique for reading data from the Internet. You'll find lots of papers and examples that use FILENAME URL for the same job in fewer lines of code, but PROC HTTP is more robust. It runs faster, and it allows you to separate the step of fetching the file from the subsequent steps of processing that file.

You can see the contents of the CSV file at this friendly URL: https://github.com/zonination/perceptions/blob/master/probly.csv. But that's not the URL that I need for PROC HTTP or any programmatic access. To download the file via a script, I need the "Raw" file URL, which I can access via the Raw button on the GitHub page.

GitHub preview

In this case, that's https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv. Here's the PROC HTTP step to download the CSV file into a temporary fileref.

/* Fetch the file from the web site */
filename probly temp;
proc http
 url="https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv"
 method="GET"
 out=probly;
run;

A note for SAS University Edition users: this step won't work for you, as the free software does not support access to secure (HTTPS) sites. You'll have to manually download the file via your browser and then continue with the remaining steps.

Step 2. Import the data into SAS with PROC IMPORT

SAS can process data with nonstandard variable names, including names that contain spaces and special characters. You simply have to use the VALIDVARNAME= system option to put SAS into the right mode (oops, almost wrote "mood" there but it's sort of the same thing).

With 'crime against nature'n.)

For this step, I'll set VALIDVARNAME=ANY to allow PROC IMPORT to retain the original column names from the CSV file. The same trick would work if I was importing from an Excel file, or any other data source that was a little more liberal in its naming rules.

/* Tell SAS to allow "nonstandard" names */
options validvarname=any;
 
/* import to a SAS data set */
proc import
  file=probly
  out=work.probly replace
  dbms=csv;
run;

Step 3. Create RENAME and LABEL statements with PROC SQL

This is one of my favorite SAS tricks. You can use PROC SQL SELECT INTO to create SAS programming statements for you, based on the data you're processing. Using this technique, I can build the parts of the LABEL statement and the RENAME statement dynamically, without knowing the variable names ahead of time.

The LABEL statement is simple. I'm going to build a series of assignments that look like this:

  'original variable name'n = 'original variable name'

I used the SELECT INTO clause to build a label assignment for each variable name. I used the CAT function to assemble the label assignment piece-by-piece, including the special literal syntax, the variable name, the assignment operator, and the label value within quotes. I'm fetching the variable names from SASHELP.VCOLUMN, one of the built-in dictionary tables that SAS provides to surface table and column metadata.

  select cat("'",trim(name),"'n","=","'",trim(name),"'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";

Here's part of the value of &labelStmt:

'Almost Certainly'n='Almost Certainly' 
'Highly Likely'n='Highly Likely' 
'Very Good Chance'n='Very Good Chance' 
'Probable'n='Probable' 
'Likely'n='Likely' 
'Probably'n='Probably' 
'We Believe'n='We Believe' 

The RENAME statement is a little trickier, because I have to calculate a new valid variable name. For this specific data source that's easy, because the only SAS "rule" that these column names violate is the ban on space characters. I can create a new name by using the COMPRESS function to remove the spaces. To be a little safer, I used the "kn" modifier on the COMPRESS function to keep only English letters, numbers, and underscores. That should cover all cases except for variable names that are too long (greater than 32 characters) or that begin with a number (or that don't contain any valid characters to begin with).

Some of the column names are one-word names that are already valid. If I include those in the RENAME statement, SAS will generate an error (you cannot "rename" a variable to its current name). I used the

/* Generate new names to comply with SAS rules.                          */
/* Assumes names contain spaces, and can fix with COMPRESS               */
/* Other deviations (like special chars, names that start with a number) */
/* would need different adjustments                                      */
/* NVALID() function can check that a name is a valid V7 name           */
proc sql noprint;
 
  /* retain original names as labels */
  select cat("'",trim(name),"'n","=","'",trim(name),"'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";
 
  select cat("'",trim(name),"'n","=",compress(name,,'kn')) 
     into :renameStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK"
  /* exclude those varnames that are already valid */
  AND not NVALID(trim(name),'V7');
quit;

Step 4. Modify the data set with new names and labels using PROC DATASETS

With the body of the LABEL and RENAME statements built, it's time to plug them into a PROC DATASETS step. PROC DATASETS can change data set attributes such as variable names, labels, and formats without requiring a complete rewrite of the data -- it's a very efficient operation.

I include the LABEL statement first, since it references the original variable names. Then I include the RENAME statement, which changes the variable names to their new V7-compliant values.

Finally, I reset the VALIDVARNAME= option to the normal V7 sanity. (Unless you're running in SAS Enterprise Guide, in which case the option is already set to ANY by default. Check this blog post for a less disruptive method of setting/restoring options.)

proc datasets lib=work nolist ;
  modify probly / memtype=data;
  label &labelStmt.;
  rename &renameStmt.;
  /* optional: report on the var names/labels */
  contents data=probly nodetails;
quit;
 
/* reset back to the old rules */
options validvarname=v7;

Here's the CONTENTS output from the PROC DATASETS step, which shows the final variable attributes. I now have easy-to-code variable names, and they still have their descriptive labels. My data dictionary dreams are coming true!

DATASETS rename output

Download the entire program example from my public Gist: import_renameV7.sas.

The post How to download and convert CSV files for use in SAS appeared first on The SAS Dummy.

5月 082017
 

In his recent article Perceptions of probability, Rick Wicklin explores how vague statements about "likeliness" translate into probabilities that we can express numerically. It's a fun, informative post -- I recommend it! You'll "Almost Certainly" enjoy it.

To prepare the article, Rick first had to download the source data from the study he cited. The data was shared as a CSV file on GitHub. Rick also had to rename the variables (column names) from the data table so that they are easier to code within SAS. Traditionally, SAS variable names must adhere to a few common programming rules: they must be alphanumeric, begin with a letter, and contain no spaces or special characters. The complete rules are documented in the this method for reading data from a cloud service like DropBox and GitHub. It's still my favorite technique for reading data from the Internet. You'll find lots of papers and examples that use FILENAME URL for the same job in fewer lines of code, but PROC HTTP is more robust. It runs faster, and it allows you to separate the step of fetching the file from the subsequent steps of processing that file.

You can see the contents of the CSV file at this friendly URL: https://github.com/zonination/perceptions/blob/master/probly.csv. But that's not the URL that I need for PROC HTTP or any programmatic access. To download the file via a script, I need the "Raw" file URL, which I can access via the Raw button on the GitHub page.

GitHub preview

In this case, that's https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv. Here's the PROC HTTP step to download the CSV file into a temporary fileref.

/* Fetch the file from the web site */
filename probly temp;
proc http
 url="https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv"
 method="GET"
 out=probly;
run;

A note for SAS University Edition users: this step won't work for you, as the free software does not support access to secure (HTTPS) sites. You'll have to manually download the file via your browser and then continue with the remaining steps.

Step 2. Import the data into SAS with PROC IMPORT

SAS can process data with nonstandard variable names, including names that contain spaces and special characters. You simply have to use the VALIDVARNAME= system option to put SAS into the right mode (oops, almost wrote "mood" there but it's sort of the same thing).

With 'crime against nature'n.)

For this step, I'll set VALIDVARNAME=ANY to allow PROC IMPORT to retain the original column names from the CSV file. The same trick would work if I was importing from an Excel file, or any other data source that was a little more liberal in its naming rules.

/* Tell SAS to allow "nonstandard" names */
options validvarname=any;
 
/* import to a SAS data set */
proc import
  file=probly
  out=work.probly replace
  dbms=csv;
run;

Step 3. Create RENAME and LABEL statements with PROC SQL

This is one of my favorite SAS tricks. You can use PROC SQL SELECT INTO to create SAS programming statements for you, based on the data you're processing. Using this technique, I can build the parts of the LABEL statement and the RENAME statement dynamically, without knowing the variable names ahead of time.

The LABEL statement is simple. I'm going to build a series of assignments that look like this:

  'original variable name'n = 'original variable name'

I used the SELECT INTO clause to build a label assignment for each variable name. I used the CAT function to assemble the label assignment piece-by-piece, including the special literal syntax, the variable name, the assignment operator, and the label value within quotes. I'm fetching the variable names from SASHELP.VCOLUMN, one of the built-in dictionary tables that SAS provides to surface table and column metadata.

  select cat("'",trim(name),"'n","=","'",trim(name),"'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";

Here's part of the value of &labelStmt:

'Almost Certainly'n='Almost Certainly' 
'Highly Likely'n='Highly Likely' 
'Very Good Chance'n='Very Good Chance' 
'Probable'n='Probable' 
'Likely'n='Likely' 
'Probably'n='Probably' 
'We Believe'n='We Believe' 

The RENAME statement is a little trickier, because I have to calculate a new valid variable name. For this specific data source that's easy, because the only SAS "rule" that these column names violate is the ban on space characters. I can create a new name by using the COMPRESS function to remove the spaces. To be a little safer, I used the "kn" modifier on the COMPRESS function to keep only English letters, numbers, and underscores. That should cover all cases except for variable names that are too long (greater than 32 characters) or that begin with a number (or that don't contain any valid characters to begin with).

Some of the column names are one-word names that are already valid. If I include those in the RENAME statement, SAS will generate an error (you cannot "rename" a variable to its current name). I used the

/* Generate new names to comply with SAS rules.                          */
/* Assumes names contain spaces, and can fix with COMPRESS               */
/* Other deviations (like special chars, names that start with a number) */
/* would need different adjustments                                      */
/* NVALID() function can check that a name is a valid V7 name           */
proc sql noprint;
 
  /* retain original names as labels */
  select cat("'",trim(name),"'n","=","'",trim(name),"'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";
 
  select cat("'",trim(name),"'n","=",compress(name,,'kn')) 
     into :renameStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK"
  /* exclude those varnames that are already valid */
  AND not NVALID(trim(name),'V7');
quit;

Step 4. Modify the data set with new names and labels using PROC DATASETS

With the body of the LABEL and RENAME statements built, it's time to plug them into a PROC DATASETS step. PROC DATASETS can change data set attributes such as variable names, labels, and formats without requiring a complete rewrite of the data -- it's a very efficient operation.

I include the LABEL statement first, since it references the original variable names. Then I include the RENAME statement, which changes the variable names to their new V7-compliant values.

Finally, I reset the VALIDVARNAME= option to the normal V7 sanity. (Unless you're running in SAS Enterprise Guide, in which case the option is already set to ANY by default. Check this blog post for a less disruptive method of setting/restoring options.)

proc datasets lib=work nolist ;
  modify probly / memtype=data;
  label &labelStmt.;
  rename &renameStmt.;
  /* optional: report on the var names/labels */
  contents data=probly nodetails;
quit;
 
/* reset back to the old rules */
options validvarname=v7;

Here's the CONTENTS output from the PROC DATASETS step, which shows the final variable attributes. I now have easy-to-code variable names, and they still have their descriptive labels. My data dictionary dreams are coming true!

DATASETS rename output

Download the entire program example from my public Gist: import_renameV7.sas.

The post How to download and convert CSV files for use in SAS appeared first on The SAS Dummy.

5月 062017
 

As SAS Viya has been gaining awareness over the past year among SAS users, there has been a lot of discussion about how SAS’ Cloud Analytic Server (CAS) handles memory vs SAS’ previous technologies such as LASR and HPA.  Recently, while I was involved in delivering several SAS Viya enablement sessions, I realised that many, including myself, held an incorrect understanding of how this works, mainly around one particular CAS option called maxTableMem.

The maxTableMem option determines the memory block size that is used per table, per CAS Worker, before converting data to memory-mapped memory.  It is not intended to directly control how much data is put into memory vs how much is put into CAS_DISK_CACHE, but rather it indirectly influences this.

Let’s unpack that a bit and try to understand what it really means.

The CAS Controller doesn’t care what the value of maxTableMem is.  In a serial load example, the CAS Controller distributes the data evenly across the CAS Workers[1], which then fill up maxTableMem-sized buckets (memory blocks), emptying them (converting them to memory-mapped memory) as they fill up, only leaving non-full buckets of table data.  You should almost never  change the default setting of this option (16MB), except perhaps in cases of extremely large tables, in order to reduce the number of file handles (up to 256MB is probably sufficient in these cases).

CAS takes advantage of standard memory mapping techniques for the CAS_DISK_CACHE, and leaves the optimisation of it up to the OS.  With SASHDAT files and LASR in SAS 9.4, the SASHDAT file essentially acts as a pre-paged file, written in a memory-mapped format, so the table data in memory doesn’t need to be written to disk when it is paged out.  Should a table need to be dropped from memory to make room for other data, and subsequently needed to be read back in to memory, it would be paged in from the SASHDAT file.

With CAS, the CAS_DISK_CACHE allows us to extend this pre-paged file approach to all data sources, not just SASHDAT.  Traditional OS swap files are written to each time memory is paged out, however with CAS, regardless of the data source (SASHDAT, database, client-uploaded file etc.) most table memory will never need to be written to disk, as it will already exist in the backing store (this could be CAS_DISK_CACHE, HDFS or NFS).   Although data will be continually paged in and out of memory, the amount of writing to disk will be minimised, which is typically slower than reading data from disk.

Another advantage of the CAS_DISK_CACHE is that when data does need to be written to disk it can happen upfront when the server is less busy, rather than at the last moment when the system detects it is out of memory (pre-paging rather than demand-paging).  Once it is written, it can be paged back into memory multiple times, by multiple concurrent processes.  The CAS_DISK_CACHE also spreads the I/O across multiple devices and servers as opposed to a typical OS swap file that may only write to a single file on a single server.

While CAS supports exceeding memory capacity by using CAS_DISK_CACHE as a backing store, read/write disk operations do have a performance cost.  Therefore, for best performance, we recommend you have enough memory capacity to hold your  most commonly used tables, meaning  most of the time the entire table will be both in memory and the backing store.

If you expect to regularly exceed memory capacity, and therefore are frequently paging data in from CAS_DISK_CACHE, consider spreading the CAS_DISK_CACHE location across multiple devices and using newer solid state storage technologies in order to improve performance.[1]

Additionally, when you need CAS to peacefully co-exist with other applications that are sharing resources on the same nodes, standard Linux cgroup settings along with Hadoop YARN configuration can be utilised to control the resources that CAS sessions can exploit.

References

Paging

Notes

[1] There are exceptions to data being evenly distributed across the CAS Workers.  The main one is if the data is partitioned and the partitions are of different sizes – all the data of a partition must be on the same node therefore resulting in an uneven distribution.  Also, if a table is very small, it may end up on only a single node, and when CAS is co-located with Hadoop the data is loaded locally from each node, so CAS receives whatever the distribution of data is that Hadoop provides.

[2] A comprehensive analysis of all possible storage combinations and the impact on performance has not yet been completed by SAS.

Dr. StrangeRAM or: How I learned to stop worrying and love CAS was published on SAS Users.

5月 052017
 
learning deep learning with keras http://p.migdal.pl/2017/04/30/teaching-deep-learning.html
移动视频2017年用户画像和趋势预测 http://mp.weixin.qq.com/s?src=3&timestamp=1493786539&ver=1&signature=LmxQAN5pURJyKafpyA7pOMD85zwzyCuxMHTKCKqXVC7D-a*-DWOtWapMbH0LA6VWonKHQy1pAp*EX0bRu5lpiTELozDJCDoTiZL4*5LVI2sMLN5DECkwAslE1rtyOmOs8zc8opBzTAfPs7sN9uqgLVhn20e-HC1lJG7SBSj*gmQ=
https://chrisalbon.com/ Notes on Data Science, Machine Learning, & Artificial Intelligence
TensorFlow template application for deep learning https://github.com/tobegit3hub/deep_recommend_system.git
Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website. 🔞  https://github.com/xiyouMc/WebHubBot
QuestionAnsweringSystem是一个Java实现的人机问答系统,能够自动分析问题并给出候选答案 https://github.com/ysc/QuestionAnsweringSystem
python 新闻联播 https://github.com/maxiee/MyCodes/blob/master/PythonJiebaProjects/XWLB_words_freq/xwlb_jieba.py
2nd place solution for the 2017 national datascience bowl http://juliandewit.github.io/kaggle-ndsb2017/
feature hash https://github.com/wush978/FeatureHashing
A framework for training and evaluating AI models on a variety of openly available dialog datasets https://github.com/facebookresearch/ParlAI


 
 Posted by at 10:25 下午