In addition to the authentication method, we can also configure access by connection type, client IP address range, and user name. I like how MySQL has the ability to only allow a user to connect to the database if he or she is logging on from a particular machine. With Vertica we can do the same thing.

What’s cool is that the Vertica database can have multiple authentication settings where each is defined as a record (single line) in the vertica.conf configuration file.

A configuration record has the following format:

ClientAuthentication = connection_type user_name address method

• The connection_type can be local, host, hostssl or hostnossl
• The user_name can be all, or one or more user names (multiple user names are separated with a plus sign (+))
• The address identifies the client machine IP address range (This can be a range)
• The method identifies the authentication method to be used for clients that match this record. The available methods are trust, reject, krb5, gss, ldap, md5 and password.

Try not to edit the vertica.conf file manually because Veritca won’t be able to error check the entries. It is recommended that we enter records into the vertica.conf file via the admintools utility (under the Configuration Menu/Edit Authentication selection) or by setting the ClientAuthentication system parameter directly.

As an example, say we want to limit user ABC so that he has to use a TCP/IP connection from his office PC (10.255.15.12) using his LDAP user name and password. That entry might look like this:

ClientAuthentication = host ABC 10.255.15.12/24 ldap “ldap://ldap.mycompany.com/;cn=;,dc=sales,dc=com”

We can also not allow user ABC to connect from a specific client machine (10.255.20.13). That entry might look like this:

ClientAuthentication = host ABC 10.255.20.13/24 reject

Note that once entries are added to the vertica.conf file the default user name and password authentication method will no longer function. To enable it in addition to our new settings we have to add a record for it in this format for TCP/IP connections:

ClientAuthentication = host all 0.0.0.0/0 password

And like this for local connections:

ClientAuthentication = local all password

When the changes are saved Vertica will check the formatting of each record and ask us to restart the database if everything checks out. Once the database is restarted we can verify that the changes to the authentication settings were successful by querying the ClientAuthentication system parameter.

dbadmin=> SELECT GET_CONFIG_PARAMETER('ClientAuthentication');
GET_CONFIG_PARAMETER
------------------------------------------------------------------------------------------------------------------------------------------------------------------
host ABC 10.255.15.12/24 ldap "ldap://ldap.mycompany.com/;cn=;,dc=sales,dc=com",host ABC 10.255.16.127/24 password,host all 0.0.0.0/0 password,local all password

The order of rules in the file is very important because Vertica will scan the entries from top to bottom using the first rule that matches the client connection. Make sure to put rules based on a specific user or IP address prior  to the more broad rule like all users or a range of IP addresses.

We can also configure the authentication options by setting the ClientAuthentication system parameter.

dbadmin=> SELECT SET_CONFIG_PARAMETER('ClientAuthentication', 'host ABC 10.255.15.12/24 ldap "ldap://ldap.mycompany.com/;cn=;,dc=sales,dc=com",host ABC 10.255.16.127/24 password,<em>host all 0.0.0.0/0 password</em>,local all password');
 SET_CONFIG_PARAMETER
 ----------------------------
 Parameter set successfully
 (1 row)

After setting the ClientAuthentication system parameter with the SET_CONFIG_PARAMETER function Vertica will automatically modify the vertica.conf file with our new settings for us. The benefit of doing it this way is that Vertica still checks our formatting and we don’t have to restart the database!

To reset the authentications method back to the default (user name and password), simply set the ClientAuthentication system parameter to nothing:

dbadmin=> SELECT SET_CONFIG_PARAMETER('ClientAuthentication', '');
SET_CONFIG_PARAMETER
----------------------------
Parameter set successfully
(1 row)

For more detail on setting up client authentication, refer to the Administrator’s Guide!

It turns out we have some mopping up to do!

Here is a simple example that reveals each of the issues. In the example I am trying to change the data type of a column from an INT to a VARCHAR.

dbadmin=> create table test1 (col1 int);
CREATE TABLE
dbadmin=> insert into test1 values (1);
OUTPUT
--------
1
(1 row)

dbadmin=> create projection test1_my_pr as select * from test1 unsegmented all nodes;
WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
CREATE PROJECTION

dbadmin=> select start_refresh();
start_refresh
----------------------------------------
Starting refresh background process.

(1 row)

dbadmin=> select projection_name
dbadmin->      , is_up_to_date
dbadmin->   from projections
dbadmin->  where anchor_table_name = 'test1';
projection_name    | is_up_to_date
----------------------+---------------
test1_node0001       | t
test1_node0002       | t
test1_node0003       | t
test1_my_pr_node0001 | t
test1_my_pr_node0002 | t
test1_my_pr_node0003 | t
(6 rows)

dbadmin=> create table test2 (col1 varchar);
CREATE TABLE
dbadmin=> insert into test2 select * from test1;
OUTPUT
--------
1
(1 row)

dbadmin=> select projection_name
dbadmin->      , is_up_to_date
dbadmin->   from projections
dbadmin->  where anchor_table_name = 'test2';
projection_name | is_up_to_date
-----------------+---------------
test2_node0001  | t
test2_node0002  | t
test2_node0003  | t
(3 rows)

dbadmin=> drop table test1 cascade;
DROP TABLE
dbadmin=> alter table test2 rename to test1;
ALTER TABLE
dbadmin=> select projection_name
dbadmin->      , is_up_to_date
dbadmin->   from projections
dbadmin->  where anchor_table_name = 'test1';
projection_name | is_up_to_date
-----------------+---------------
test2_node0001  | t
test2_node0002  | t
test2_node0003  | t
(3 rows)

Notice that the projection name on each node does not start with “test1″ and we lost the test1_my_pr projection altogether!

To resolve these issues we’re going to have to modify the “5 easy steps” that were discussed in the earlier post.

As a reminder, the “5 easy steps” were listed as:

1. Create a new table similar to original, but include the data type or size modifications
2. Copy the data from the original table to the new table
3. Rename the old table (as a backup)
4. Rename the new table to same name as the original table
5. Drop the old table (unless you want to keep it around as a backup)

To handle both issues (keeping projection names intact and rebuilding  projections) we need to add a few more steps:

Add Step 2.5:

Extract the projections for the original table before dropping it using the EXPORT_OBJECTS function. This function can generate a script file that contains all of the projection CREATE statements for our table. We can execute the script after we’ve renamed the temporary table back to the original to re-create the projections.

dbadmin=> select export_objects('/home/dbadmin/test1.sql', 'test1');
export_objects
-------------------------------------
Catalog data exported successfully

(1 row)

dbadmin=> \! cat test1.sql

CREATE TABLE public.test1
(
col1 int
);

CREATE PROJECTION public.test1
(
col1
)
AS
SELECT test1.col1
FROM public.test1
ORDER BY test1.col1
UNSEGMENTED ALL NODES;

CREATE PROJECTION public.test1_my_pr
(
col1
)
AS
SELECT test1.col1
FROM public.test1
ORDER BY test1.col1
UNSEGMENTED ALL NODES;

SELECT MARK_DESIGN_KSAFE(1);
dbadmin=> select * from test2;
col1
------
1
(1 row)

Add Step 2.6:

Edit the script file and remove the CREATE TABLE statement and add a SELECT START_REFRESH(); statement at the end to refresh the projections.

Add Step 4.5:

Run the edited script!

dbadmin=> \i /home/dbadmin/test1.sql
vsql:/home/dbadmin/test1.sql:18: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
vsql:/home/dbadmin/test1.sql:18: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
vsql:/home/dbadmin/test1.sql:18: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
CREATE PROJECTION
vsql:/home/dbadmin/test1.sql:28: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
vsql:/home/dbadmin/test1.sql:28: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
vsql:/home/dbadmin/test1.sql:28: WARNING:  Projection  is not available for query processing. Execute the select start_refresh() function to copy data into this projection.
The projection must have a sufficient number of buddy projections and all nodes must be up before starting a refresh.
CREATE PROJECTION
MARK_DESIGN_KSAFE
----------------------
Marked design 1-safe
(1 row)

START_REFRESH
----------------------------------------
Starting refresh background process.

(1 row)

dbadmin=> select projection_name
dbadmin->      , is_up_to_date
dbadmin->   from projections
dbadmin->  where anchor_table_name = 'test1';
 projection_name    | is_up_to_date
 ----------------------+---------------
 test2_node0001       | t
 test2_node0002       | t
 test2_node0003       | t
 test1_node0001       | t
 test1_node0002       | t
 test1_node0003       | t
 test1_my_pr_node0001 | t
 test1_my_pr_node0002 | t
 test1_my_pr_node0003 | t
 (9 rows)

Finally, one more step is needed to clean up the old projections that are no longer needed (those named after the temporary TEMP2 table).

Step 4.7:

Drop the old projections:

dbadmin=> drop projection test2_node0001;
DROP PROJECTION
dbadmin=> drop projection test2_node0002;
DROP PROJECTION
dbadmin=> drop projection test2_node0003;
DROP PROJECTION
dbadmin=> select projection_name
dbadmin->      , is_up_to_date
dbadmin->   from projections
dbadmin->  where anchor_table_name = 'test1';
projection_name    | is_up_to_date
----------------------+---------------
test1_node0001       | t
test1_node0002       | t
test1_node0003       | t
test1_my_pr_node0001 | t
test1_my_pr_node0002 | t
test1_my_pr_node0003 | t
(6 rows)

dbadmin=> \d test1;
List of Fields by Tables
Schema | Table | Column |    Type     | Size | Default | Not Null | Primary Key | Foreign Key
--------+-------+--------+-------------+------+---------+----------+-------------+-------------
public | test1 | col1   | varchar(80) |   80 |         | f        | f           |
(1 row)

dbadmin=> select * from test1;
col1
------
1
(1 row)

So our “5 easy steps” have grown a little into these “9 easy steps“:

1. Create a new table similar to original, but include the data type or size modifications
2. Copy the data from the original table to the new table
3. Extract the projection create statements for the original table into a script.
4. Rename the old table (as a backup)
5. Edit the script created in step 3 to remove the CREATE TABLE statement and add a SELECT START_REFRESH() statement to the end.
6. Rename the new table to same name as the original table
7. Run the script edited in step 5 to re-create the projections
8. Drop the old projections
9. Drop the old table (unless you want to keep it around as a backup)

Hopefully this helps out a little more!

First in Apache2, you can get rid of a lot of the default modules, by commenting out these modules you save about 10mb of ram per request (don’t comment out anything you need though):

LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule speling_module modules/mod_speling.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cgi_module modules/mod_cgi.so

I would recommend going to the top of the file and turning KeepAlive On too, this could save your behind if you get a big surge of traffic.

also, in your conf.d folder, rename  proxy_ajp.conf to proxy_ajp.conf.old

In MySQL, I usually run this for my default mysql instancess (2-4gb of ram) and tweak more as needed (my.cnf):

query_cache_type=1
query_cache_limit=1M
query_cache_size=32M
thread_concurrency=4
log_slow_queries=/tmp/mysqld.slow.log
long_query_time=2
key_buffer_size=32M
sort_buffer_size=10M
read_buffer_size=10M
join_buffer_size=10M
read_rnd_buffer_size=25M
max_heap_table_size=256M
tmp_table_size=256M
thread_cache=256
myisam_sort_buffer_size=128M

So we can make changes like the following using the ALTER TABLE ALTER COLUMN command:

dbadmin=> select version();
version
------------------------------------
Vertica Analytic Database v5.1.1-0
(1 row)
dbadmin=> create table test (col1 varchar(100));
CREATE TABLE
dbadmin=> alter table test alter column col1 set not null;
ALTER TABLE
dbadmin=> alter table test alter column col1 set default '0';
ALTER TABLE
dbadmin=> alter table test alter column col1 drop default;
ALTER TABLE
dbadmin=> alter table test alter column col1 drop not null;
ALTER TABLE

But not change like these:

dbadmin=> alter table test alter column col1 int;
ERROR:  syntax error at or near "int" at character 36
LINE 1: alter table test alter column col1 int;
                                           ^
dbadmin=> alter table test alter column col1 varchar(200);
ERROR:  syntax error at or near "varchar" at character 36
LINE 1: alter table test alter column col1 varchar(200);
                                           ^

As I mentioned, there are 5 steps that are needed to change a column data type or to re-size it.

They are:

1. Create a new table similar to original, but include the data type or size modifications
2. Copy the data from the original table to the new table
3. Rename the old table (as a backup)
4. Rename the new table to same name as the original table
5. Drop the old table (unless you want to keep it around as a backup)

As an example, in the health care industry the number of medical diagnosis codes (called ICD codes) increased when version 10 was introduced. In doing so the number of characters that represent a code have increased from 3-5 characters (ICD version 9) in length up to 3-7 characters in length for ICD version 10. If a field in a table that stores an ICD code is defined smaller than 7 characters it will need to be modified.

Here’s how to do that.

First let’s create a test table named ICD_CODE and I’ll insert a few ICD-9 codes.

dbadmin=> create table icd_code (version int, code varchar(5), description varchar(255));
CREATE TABLE
dbadmin=> insert into icd_code values (9, '157', 'Malignant neoplasm of pancreas');
OUTPUT
--------
1
(1 row)

dbadmin=> insert into icd_code values (9, '1570', 'Head of pancreas');
OUTPUT
--------
1
(1 row)

Next I’ll try and insert an ICD-10 code:

dbadmin=> insert into icd_code values (10, 'S52521A', 'Torus fracture of lower end of right radius, initial encounter for closed fracture');
ERROR:  7-octet value too long for type Varchar(5)

Yikes! We can’t squeeze it into the column.

We’re going to have to modify the CODE column so that it can fit the length of the new ICD. To do that, we’re going to have to implement the 5 relatively easy steps listed above…

1. Create a new table similar to original, but include the data type or size modifications

dbadmin=> create table icd_code_temp (version int, code varchar(7), description varchar(255));
CREATE TABLE

2. Copy the data from the original table to the new table

dbadmin=> insert into icd_code_temp select * from icd_code;
OUTPUT
--------
2
(1 row)

3. Rename the old table (as a backup)

dbadmin=> alter table icd_code rename to icd_code_bk;
ALTER TABLE

4. Rename the new table to same name as the original table

dbadmin=> alter table icd_code_temp rename to icd_code;
ALTER TABLE

Now we can insert the ICD-10 code:

dbadmin=> insert into icd_code values (10, 'S52521A', 'Torus fracture of lower end of right radius, initial encounter for closed fracture');
 OUTPUT
 --------
 1
 (1 row)

Verify that all the records are in the new table:

dbadmin=> select * from icd_code;
version |  code   |                                    description
---------+---------+------------------------------------------------------------------------------------
9 | 157     | Malignant neoplasm of pancreas
9 | 1570    | Head of pancreas
10 | S52521A | Torus fracture of lower end of right radius, initial encounter for closed fracture
(3 rows)

5. Drop the old table (unless you want to keep it around as a backup)

dbadmin=> drop table icd_code_bk;
DROP TABLE

Hopefully the ALTER TABLE ALTER COLUMN command in a future release can do this for us!

According to Vertica’s documentation the “Syslog is a network-logging utility that issues, stores, and processes meaningful log messages. It is designed so DBAs can keep machines up and running, and it is a useful way to get heterogeneous data into a single data repository.”

Since I’m a DBA that likes to keep Vertica up and running (and one that likes to be employed) I figured it’d be a good idea to take a look at this feature!

Basically, there are three configuration steps needed to start event logging to the syslog:

1. Turn the feature on (off by default)
2. Indicate which events to log (none by default)
3. Indicate where to post the events (/var/log/messages by default)

Note: The third step is only necessary if you want to change the facility logging the events.  Please see Vertica’s Administrator’s Guide for details on how to change the default facility for the syslog.

Enabling and disabling the posting of events to the syslog is done by setting the SyslogEnabled system parameter.

To enable use: SELECT SET_CONFIG_PARAMETER(‘SyslogEnabled’, 1);

To disable use: SELECT SET_CONFIG_PARAMETER(‘SyslogEnabled’, 0);

Here is a list of events that can be monitored:

• Low Disk Space
• Read Only File System
• Loss Of K Safety
• Current Fault Tolerance at Critical Level
• Too Many ROS Containers
• WOS Over Flow
• Node State Change
• Recovery Failure
• Recovery Error
• Recovery Lock Error
• Recovery Projection Retrieval Error
• Refresh Error
• Refresh Lock Error
• Tuple Mover Error
• Timer Service Task Error
• Stale Checkpoint

Use the SyslogEvents system parameter to let Vertica know which events you want to log. For example:

SELECT SET_CONFIG_PARAMETER(‘SyslogEvents’, ‘Low Disk Space,  Node State Change’);

By default events are posted in /var/log/messages.

As a test, I’ll tell Vertica to log any “Node State Change” events and then stop and start Vertica on node 3 of my three node cluster.

STEP 1 – Set up Vertica:

dbadmin=> SELECT SET_CONFIG_PARAMETER('SyslogEvents', 'Node State Change');
SET_CONFIG_PARAMETER
----------------------------
Parameter set successfully
(1 row)

dbadmin=> SELECT SET_CONFIG_PARAMETER('SyslogEnabled', 1);
SET_CONFIG_PARAMETER
----------------------------
Parameter set successfully
(1 row)

STEP 2 -Stop and start Vertica on node 3 using admintools utility

STEP 3 – Jump out the OS on node 3 and view the /var/log/messages file:

[root@verticatst03 ~]#  tail /var/log/messages
Apr 18 11:29:42 verticatst03 vertica: Event Posted: Event Code:6 Event Id:8 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:29:42.343087 ExpirationTimestamp: 2080-05-06 13:43:49.343087 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 startup state to SHUTDOWN DatabaseName: vtest Hostname: verticatst03
Apr 18 11:29:42 verticatst03 vertica: Event Cleared: Event Code:6 Event Id:6 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:29:42.343557 ExpirationTimestamp: 2012-04-18 11:29:42.343557 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 leaving startup state UP DatabaseName: vtest Hostname: verticatst03
Apr 18 11:32:18 verticatst03 vertica: Event Posted: Event Code:6 Event Id:1 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:32:18.051284 ExpirationTimestamp: 2080-05-06 13:46:25.051284 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 startup state to RECOVERING DatabaseName: vtest Hostname: verticatst03
Apr 18 11:32:19 verticatst03 vertica: Event Posted: Event Code:6 Event Id:7 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:32:19.686469 ExpirationTimestamp: 2080-05-06 13:46:26.686469 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 startup state to READY DatabaseName: vtest Hostname: verticatst03
Apr 18 11:32:19 verticatst03 vertica: Event Cleared: Event Code:6 Event Id:1 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:32:19.686775 ExpirationTimestamp: 2012-04-18 11:32:19.686775 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 leaving startup state RECOVERING DatabaseName: vtest Hostname: verticatst03
Apr 18 11:32:19 verticatst03 vertica: Event Posted: Event Code:6 Event Id:6 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:32:19.687075 ExpirationTimestamp: 2080-05-06 13:46:26.687075 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 startup state to UP DatabaseName: vtest Hostname: verticatst03
Apr 18 11:32:19 verticatst03 vertica: Event Cleared: Event Code:6 Event Id:7 Event Severity: Informational [6] PostedTimestamp: 2012-04-18 11:32:19.687184 ExpirationTimestamp: 2012-04-18 11:32:19.687184 EventCodeDescription: Node State Change ProblemDescription: Changing node v_vtest_node0003 leaving startup state READY DatabaseName: vtest Hostname: verticatst03

Note that I had to check /var/log/messages on my cluster node 3 because that’s the node I took down!

Now I can easily review the syslog for system errors and the Database events I choose in a single location!

The syntax for the TIME_SLICE function is:

TIME_SLICE(expression, slice_length, [ time_unit = 'SECOND' ],  [ start_or_end = 'START' ] )

The expression is either a TIMESTAMP or a (string) constant that can be parsed into a TIMESTAMP value, slice_length is the length of the slice specified, time_unit is the time unit of the slice, and start_or_end indicates whether the returned value corresponds to the start or end time of the time slice interval.

Example:

I want to see my company’s sales information as the number of sales, max sale, min sale and avg sale in 60 second intervals. For this simple example, the company sales are stored in a table named test where there is a column to store the date and time of the sale and a column to store the dollar amount of the sale:

dbadmin=> select * from test order by 1;
a_date        | sales
---------------------+-------
2012-04-10 00:00:00 |   700
2012-04-10 23:59:45 |   600
2012-04-11 00:00:00 |   300
2012-04-11 00:00:01 |   430
2012-04-11 00:00:34 |   430
2012-04-11 00:01:00 |   100
2012-04-11 00:01:02 |   150
2012-04-11 00:01:14 |   230
2012-04-11 00:02:32 |   330
2012-04-11 00:02:50 |   350
2012-04-11 00:02:51 |   150
2012-04-11 00:03:01 |   125
2012-04-11 00:03:25 |   900
2012-04-11 00:03:26 |   800
2012-04-11 00:03:26 |   900
2012-04-11 00:03:35 |   425
2012-04-11 00:04:00 |    50
2012-04-11 00:05:25 |   900
2012-04-11 00:05:30 |  5000
2012-04-11 00:05:31 |  6000
2012-04-11 00:05:31 |  6500
2012-04-11 00:05:43 |  5000
2012-04-11 00:13:03 |   300
(23 rows)

Now, using the TIME_SLICE function, I can aggregate the sales in 60 second intervals:

dbadmin=> select time_slice(a_date, 60, 'second', 'start') ts_start
dbadmin->      , time_slice(a_date, 60, 'second', 'end') ts_end
dbadmin->      , count(*) tot_sales
dbadmin->      , min(sales) min_sales
dbadmin->      , max(sales) max_sales
dbadmin->      , ROUND(avg(sales), 2) avg_sales
dbadmin->   from test
dbadmin->  group
dbadmin->     by time_slice(a_date, 60, 'second', 'start')
dbadmin->      , time_slice(a_date, 60, 'second', 'end')
dbadmin->  order by 1;
ts_start       |       ts_end        | tot_sales | min_sales | max_sales | avg_sales
---------------------+---------------------+-----------+-----------+-----------+-----------
2012-04-10 00:00:00 | 2012-04-10 00:01:00 |         1 |       700 |       700 |       700
2012-04-10 23:59:00 | 2012-04-11 00:00:00 |         1 |       600 |       600 |       600
2012-04-11 00:00:00 | 2012-04-11 00:01:00 |         3 |       300 |       430 |    386.67
2012-04-11 00:01:00 | 2012-04-11 00:02:00 |         3 |       100 |       230 |       160
2012-04-11 00:02:00 | 2012-04-11 00:03:00 |         3 |       150 |       350 |    276.67
2012-04-11 00:03:00 | 2012-04-11 00:04:00 |         5 |       125 |       900 |       630
2012-04-11 00:04:00 | 2012-04-11 00:05:00 |         1 |        50 |        50 |        50
2012-04-11 00:05:00 | 2012-04-11 00:06:00 |         5 |       900 |      6500 |      4680
2012-04-11 00:13:00 | 2012-04-11 00:14:00 |         1 |       300 |       300 |       300
(9 rows)

Have fun and have a “slice” of pie with that!

PS. Check out a very interesting TIME_SLICE forum discussion at vertica-forums.com!

Even though the Vertica Community Edition can support up to three nodes, I’m going to install a single node in this tutorial. This configuration should only be used for development. In a future article, I’ll go over installing Vertica with more than one node.

Here are the steps:

Download and Install the Vertica Rpm.

• My server is virtual and running Centos 6.0.

[root@Anvil ~]# rpm -Uvh vertica-ce-5.1.1-0.x86_64.RHEL5.rpm

Preparing… ########################################### [100%]
1:vertica-ce ########################################### [100%]

Vertica Analytic Database V5.1.1-0 successfully installed on host Anvil

———————————————————————————-
Important Information
———————————————————————————-
If you are upgrading from a previous version, you must backup your database before
continuing with this install. After restarting your database, you will be unable
to revert to a previous version of the software.
———————————————————————————-

To download the latest Vertica documentation in zip or tar format please visit the
myvertica web site.

To complete installation and configuration of the cluster,
run: /opt/vertica/sbin/install_vertica

[root@Anvil ~]#

Run install_vertica Script

• We are installing the single node instance.  Please remember that you will not be able expand this in the future

[root@Anvil ~]# /opt/vertica/sbin/install_vertica

Vertica Analytic Database 5.1.1-0 Installation Tool
Upgrading admintools meta data format..
scanning /opt/vertica/config/users
WARNING: No hostname list provided. Installing to localhost
Starting installation tasks…
Getting system information for cluster (this may take a while)….
Info: the package ‘pstack’ is useful during troubleshooting. Vertica recommends this package is installed.
Info: the package ‘sysstat’ is useful during troubleshooting. Vertica recommends this package is installed.
Checking/fixing OS parameters…..

Info! Parameter vm.min_free_kbytes is less than 4096
Setting vm.min_free_kbytes to 4096 …
Info! The maximum number of open file descriptors is less than 65536
Setting open filehandle limit to 65536 …
Info! The session setting of pam_limits.so is not set in /etc/pam.d/su
Setting session of pam_limits.so in /etc/pam.d/su …
Info! Parameter fs.file-max is less than 65536
Setting fs.file-max to 65536 …
Detected cpufreq module loaded on localhost
CPU frequency scaling is enabled. This may adversely affect the performance of your database.
Vertica recommends that cpu frequency scaling be turned off or set to ‘performance’
Creating/Checking Vertica DBA group

Creating/Checking Vertica DBA user

Password for dbadmin:

Creating Vertica Data Directory…

Updating spread configuration…
Verifying spread configuration on whole cluster
Generating RSA private key, 1024 bit long modulus
……..++++++
………………………………………………….++++++
e is 65537 (0×10001)
Creating node node0001 definition for host localhost
… Done
Error Monitor 0 errors 2 warnings
Installation completed with warnings.
Installation complete.

To create a database:
1. Logout and login as dbadmin.**
2. Run /opt/vertica/bin/adminTools as dbadmin
3. Select Create Database from the Configuration Menu

** The installation modified the group privileges for dbadmin.
If you used sudo to install vertica as dbadmin, you will
need to logout and login again before the privileges are applied.

[root@Anvil ~]#

We’re ready to start playing Vertica!
This is a vanilla installation. The install_vertica script has a number of parameters, but they are unnecessary for this basic installation. Please refer back to to Vertica’s documentation to review these parameters.  Please note, you will NOT be able to add additional nodes to this install!  You will have to reinstall to support additional nodes.  So please, only use this method for prototyping.

As I mentioned in my original article, I’ve used a number of different databases.  I decided that I wanted to document my work with MongoDB.  This MongoDB tutorial contains a straight forward guide to installing MongoDB on a Centos server.

You can download the files which are created in this tutorial from here.

1. Create Data Directory for MongoDB

• The MongDB Quickstart guide for Linux recommends creating a /data/db/ directory. I prefer to follow the Filesystem Hierarchy Standard (FHS) and place the data files in /var/lib.

[root@Anvil ~]# mkdir /var/lib/mongodb/
[root@Anvil ~]# chown -R mongodb:mongodb /var/lib/mongodb/

2. Create the MongoDB User

• Create a user to run MongoDB. DO NOT RUN IT AS ROOT!!!

  [root@Anvil ~]# groupadd mongodb
  [root@Anvil ~]# useradd -r -g mongodb mongodb
  

  3. Download MongoDB from Here

  • I just download it into the root directory.

  [~]# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.4.tgz

  4. Extract the files

  [root@Anvil ~]# tar xzf mongodb-linux-x86_64-2.0.4.tgz

  [root@Anvil ~]# ll mongodb-linux-x86_64-2.0.4
  total 56
   drwxr-xr-x 2 root root  4096 Mar 20 00:21 bin
  -rw------- 1 root root 34520 Feb 13 16:05 GNU-AGPL-3.0
  -rw------- 1 root root  1364 Feb 13 16:05 README
  -rw------- 1 root root  9879 Feb 13 16:05 THIRD-PARTY-NOTICES

  5. Copy the /bin Files to /usr/bin

  [root@Anvil ~]# cp mongodb-linux-x86_64-2.0.4/bin/* /usr/bin/

  6. Copy Remaining Files to /var/lib/mongodb/

  [root@Anvil ~]# cp mongodb-linux-x86_64-2.0.4/* /var/lib/mongodb/

  7. Create MongoDB Start and Stop Scripts

  • Create basic shell scripts to stop and start the database.
  • These scripts should be copied to /usr/bin/.

  [root@Anvil ~]# vi /usr/bin/mongostart
  
  #!/bin/sh
  
  /usr/bin/mongod --config /etc/mongodb.cnf \
  ## --iupgrade \ ##runs a database upgrade option if needed \
  
  [root@Anvil ~]# vi /usr/bin/mongostop
  
  #!/bin/bash
  
  pid=`'ps -o pid,command ax | grep mongod | awk '!/awk/ && !/grep/ {print $1}'`;
  if [ "${pid}" != "" ]; then
      kill -2 ${pid};
  fi
  

  8. Create MongoDB Configuration File

  • I always create a basic configuration file for MongoDB in /etc.

  [root@Anvil ~]# vi /etc/mongodb.cnf
  
  # Configuration Options for MongoDB
  
  dbpath = /var/lib/mongodb/
  logpath = /var/lib/mongodb/mongodb.log
  logappend = true
  
  bind_ip = 127.0.0.1
  port = 27017
  fork = true
  auth = true

  9. Create MongoDB Init File

  [root@Anvil ~]# vi /etc/mongodb.cnf
  
  #!/bin/sh
  #
  # mongodb – this script starts and stops the mongodb daemon
  #
  # chkconfig: - 85 15
  # description: MongoDB is a non-relational database storage system.
  # processname: mongodb
  # config: /etc/mongodb.cnf
  # pidfile: /var/lib/mongodb/mongo.pid
  
  PATH=/sbin:/bin:/usr/sbin:/usr/bin
  NAME=mongodb
  
  test -x $DAEMON || exit 0
  
  set -e
  
  case "$1" in
    start)
          echo -n "Starting MongoDB... "
          su mongodb -c "/usr/bin/mongostart"
          ;;
    stop)
          echo -n "Stopping MongoDB"
          /usr/bin/mongostop
          ;;
        *)
              N=/etc/init.d/$NAME
              echo "Usage: $N {start|stop}" >&2
              exit 1
              ;;
      esac
  
      exit 0
  

  • Save as /etc/init.d/mongodb.

  [root@Anvil ~]# chmod +x /etc/init.d/mongodb
  [root@Anvil ~]# chkconfig --add mongodb
  [root@Anvil ~]# chkconfig mongodb on
  
  Starting MongoDB... [root@Anvil lib]# forked process: 28108
  all output going to: /var/lib/mongodb/mongodb.log

I’ll use the TIMESTAMPDIFF function in my examples. This function has the general specification:

TIMESTAMPDIFF ( datepart , starttimestamp , endtimestamp );

So if I want to know the number of hours that occur between 8am and 4pm on April 2, 2012 I could do this:

dbadmin=> SELECT TIMESTAMPDIFF ('HOUR',('apr 2, 2012 08:00:00'), ('apr 2, 2012 16:00:00'));
timestampdiff
---------------
8
(1 row)

That looks right. How about between 8am and 4:30pm on April 2, 2012?

dbadmin=> SELECT TIMESTAMPDIFF ('HOUR',('apr 2, 2012 08:00:00'), ('apr 2, 2012 16:30:00'));
timestampdiff
---------------
8
(1 row)

Hmm. Sure there is an 8 hour difference in the dates, but what about the extra 30 minutes? 8 hours was not the result the user was expecting. He was hoping to see 8.5 hours.

What I found is that the date diff functions in Vertica work in “ticks” of time. That is, they operate on whole increments of the “datepart” specified. In my case, I used “HOUR” so I got only whole hours back. The same is true when using the other “datepart” indicators passed into the functions.

For instance, how many months are there between April 2, 2012 at 8am and May 15, 2012 at 8 am?

dbadmin=> SELECT TIMESTAMPDIFF ('MONTH',('apr 2, 2012 08:00:00'), ('may 15, 2012 08:00:00'));
timestampdiff
---------------
1
(1 row)

Since we specified months Vertica is only comparing the months and disregarding the fractional part of the month. There is 1 month between April and May. (Hint: If you need to find the exact number of months between two dates use the MONTHS_BETWEEN function!)

Back to our original requirement, how can we get the number of hours including fractional parts of an hour in the result?

For that we have to get more granular and find the number of seconds that occurred between the two dates and then convert them back to hours (divide by 3600 which is the number of seconds in an hour):

dbadmin=> SELECT TIMESTAMPDIFF ('SECOND',('apr 2, 2012 08:00:00'), ('apr 2, 2012 16:30:00')) / 3600.0 as diff;
diff
-------------------
8.500000000000000
(1 row)

Now the user is content.

Notice that I divided by 3600.0 and not 3600. In Vertica if we divide by an integer we’ll get an integer result (rounded to an integer value). If we add a “.0″ to the divisor then we are now dividing an integer by a float, which returns a float!.

Here’s what happens if I divide by 3600:

dbadmin=> SELECT TIMESTAMPDIFF ('SECOND',('apr 2, 2012 08:00:00'), ('apr 2, 2012 16:30:00')) / 3600 as diff;
diff
------
8
(1 row)

Here are a few more examples:

dbadmin=> SELECT TIMESTAMPDIFF ('SECOND',('apr 2, 2012 08:00:00'), ('apr 2, 2012 08:30:00')) / 3600.0 as diff;
diff
------------------
.500000000000000
(1 row)

dbadmin=> SELECT TIMESTAMPDIFF ('SECOND',('apr 2, 2012 08:00:00'), ('apr 3, 2012 08:30:00')) / 3600.0 as diff;
diff
--------------------
24.500000000000000
(1 row)

dbadmin=> SELECT TIMESTAMPDIFF ('SECOND',('apr 3, 2012 08:40:00'), ('apr 3, 2012 08:50:00')) / 3600.0 as diff;
diff
------------------
.166666666666667
(1 row)

Check out the Vertica Forums for the latest and greatest Vertica discussion topics!

Here is a quick example using runs scored by three MLB teams in the 2008, 2009, 2010 and 2011 seasons (Note: The scores used are made up because I was too lazy to Google actuals).

First I’ll create a table named MLB and insert some data:

dbadmin=>; create table mlb (season INT, team varchar(100), runs int);
CREATE TABLE

dbadmin=> insert into mlb values (2008, 'Pirates', 350);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2009, 'Pirates', 325);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2010, 'Pirates', 296);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2011, 'Pirates', 301);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2008, 'Yankees', 450);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2009, 'Yankees', 487);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2010, 'Yankees', 387);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2011, 'Yankees', 404);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2008, 'Dodgers', 250);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2009, 'Dodgers', 347);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2010, 'Dodgers', 237);
OUTPUT
--------
1
(1 row)

dbadmin=> insert into mlb values (2011, 'Dodgers', 544);
OUTPUT
--------
1
(1 row)

If I select directly from the MLB table, my result set looks like this:

dbadmin=>SELECT * FROM mlb;
season |  team   | runs
--------+---------+------
2008 | Pirates |  350
2009 | Pirates |  325
2010 | Pirates |  296
2011 | Pirates |  301
2008 | Yankees |  450
2009 | Yankees |  487
2010 | Yankees |  387
2011 | Yankees |  404
2008 | Dodgers |  250
2009 | Dodgers |  347
2010 | Dodgers |  237
2011 | Dodgers |  544
(12 rows)

I’d like to pivot my results on the SEASON column so that I can see the runs scored in columns as opposed to in the rows. To do that I can use the DECODE function:

dbadmin=> SELECT team
dbadmin->           , DECODE(season, 2008, runs, NULL) AS "2008_Season"
dbadmin->           , DECODE(season, 2009, runs, NULL) AS "2009_Season"
dbadmin->           , DECODE(season, 2010, runs, NULL) AS "2010_Season"
dbadmin->           , DECODE(season, 2011, runs, NULL) AS "2011_Season"
dbadmin->        FROM (SELECT season
dbadmin->                   , team
dbadmin->                   , runs
dbadmin->                FROM mlb) seasons;
team   | 2008_Season | 2009_Season | 2010_Season | 2011_Season
---------+-------------+-------------+-------------+-------------
Pirates |         350 |             |             |
Pirates |             |         325 |             |
Pirates |             |             |         296 |
Pirates |             |             |             |         301
Yankees |         450 |             |             |
Yankees |             |         487 |             |
Yankees |             |             |         387 |
Yankees |             |             |             |         404
Dodgers |         250 |             |             |
Dodgers |             |         347 |             |
Dodgers |             |             |         237 |
Dodgers |             |             |             |         544
(12 rows)

We’re not finished yet as we need to get rid of the NULL column values. For that, we use the MAX function:

dbadmin=> SELECT team
dbadmin->      , MAX(DECODE(season, 2008, runs, NULL)) AS "2008_Season"
dbadmin->      , MAX(DECODE(season, 2009, runs, NULL)) AS "2009_Season"
dbadmin->      , MAX(DECODE(season, 2010, runs, NULL)) AS "2010_Season"
dbadmin->      , MAX(DECODE(season, 2011, runs, NULL)) AS "2011_Season"
dbadmin->   FROM (SELECT season
dbadmin->              , team
dbadmin->              , runs
dbadmin->           FROM mlb) seasons
dbadmin->  GROUP
dbadmin->     BY team;
team   | 2008_Season | 2009_Season | 2010_Season | 2011_Season
---------+-------------+-------------+-------------+-------------
Dodgers |         250 |         347 |         237 |         544
Yankees |         450 |         487 |         387 |         404
Pirates |         350 |         325 |         296 |         301
(3 rows)

That’s it! This is an old trick we used in Oracle before they introduced the PIVOT SQL command in 11g. Maybe Vertica will introduce a similar feature! HINT!