Showing posts with label big data. Show all posts
Showing posts with label big data. Show all posts

Tuesday, June 19, 2012

Cassandra quick notes (Part I)

I had a number of quick notes on Cassandra, which I thought others might find useful as well. Since my original set of quick notes is pretty long, I am breaking it up into two parts (this is the first part). If you are interested in the "operations" aspects of Cassandra, I would recommend looking at this book; a lot of the pointers are from this book.
  • Calculate ideal initial tokens
$  \text{init_token} = \text{node_num_zero_indexed} \times \frac{2^{127}}{\text{num_nodes}} $
  • Adjust replication factor to work with quorum
$ \text{nodes_for_quorum} = \frac{\text{rep_factor}}{2} + 1 $
  • Anti Entropy Repair
Anti Entropy Repair (AES) is a very intensive data repair mechanism, and should preferably be run at times of low traffic. It can result in duplicate data on nodes, which can be removed using nodetool compact, or can be eliminated during the repair process using a the $\text{-pr}$ option. Schedule for AES should be lower or equal to $\text{gc_grace_seconds}$. AES should be run in the following situations:
    • Change in replication factor
    • Joined nodes without auto bootstrap
    • Lost or corrupted fils (such as SSTables, indexes, or commit logs)
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT repair -pr
  • Nodetool cleanup
Use nodetool cleanup to remove copies of data from nodes for which they are not
responsible for. Cleanup is intensive; run cleanup for topology changes, or
hinted handoff and write consistency ANY.
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT cleanup
  • Use nodetool snapshot for backup
Snapshot makes hard-links of files in the data directory to a subfolder,
"snapshot/<timestamp>"
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT snapshot
  • Clear snapshots with nodetool
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT clearsnapshot
  • Nodetool to move nodes in ring
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT move <new_token>
  • Nodetool to remove a "downed" node
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT removetoken <token_value>
When a node is removed, Cassandra actively begins replicating the missing data until it is stored on the number of nodes specified by replication factor.
  • Removing a "live" node
$ <cassandra_home>/bin/nodetool -h $LIVE_NODE_TO_REMOVE -p $PORT decommission
  • Get quick stats using nodetool
The following are pretty handy in quickly gleaming at Cassandra cluster state.
$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT tpstats

$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT cfstats

$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT compactionstats

$ <cassandra_home>/bin/nodetool -h $HOST -p $PORT cfhistgrams
  • Monitor GC events in Cassandra log
Cassandra has options in $\text{conf/cassandra-env.sh}$ that cause Java to print
garbage collection message to log file.
$ grep "GC inspection" /var/log/cassandra/system.log

Wednesday, March 28, 2012

More data, more noise, or more rare events

We are in a data splurge. Everyone is interested in data, how to gather it efficiently, how to store it, and most importantly, what to make out of it. Data is playing a key part from search, ad and movie recommendation, to development of social media based products. We want to know more and more about our users, be more personalized. So we collect more data, in an attempt to cover every aspect of our users' likes, preferences and habits.

But how does one decide what data is worth collecting? Or, do we just collect everything we can get our hands onto? How does one find the balance, between collecting noisy data and those informative events that will give us the crucial insight?

Friday, March 23, 2012

Setting up a local hadoop cluster

I am running a hadoop cluster on my local machine, for development purposes, in a pseudo distributed mode. In pseudo distributed mode, hadoop daemons run in separate processes.
Just thought I will outline the basic setup process here, in case someone else finds it handy. It is actually quite straightforward to get a hadoop cluster running locally. Here are the basic steps:
  • You need to download a hadoop implementation. The one I am using is from Cloudera, and you can obtain a tarball from here.
  • Expand the tarball into some convenient directory on your local machine. You will find a bin directory which contains all necessary files for managing hadoop, and a conf directory where hadoop configurations are located. You will have to modify conf-site.xml, hdfs-site.xml, and mapred-site.xml to specify name nodes, replications, etc. to customize your cluster setup. More on this at the bottom.
  • You need to ensure remote login, that is, check if you can do ssh localhost without being asked for a password. If you already have a public key, then you can run the following command (assuming dsa key) to ensure password-less local ssh login.

    cat $HOME/.ssh/id_dsa.pub >> $HOME/.ssh/authorized_keys

  • Set the following environment variables:

    HADOOP_HOME: Path to hadoop directory

    HADOOP_CONF_DIR: Path to hadoop conf directory

    Also add the hadoop bin directory in you path.

  • Create a hadooptemp directory (see below in core-site.xml) using the following command:

    sudo mkdir -m 777 hadooptemp

  • Format the data node:

    hadoop namenode -format

  • Bring up pseudo mode by invoking start-all.sh. You can stop pseudo mode using stop-all.sh.
You can set up override configurations for hadoop following the examples below.

An example core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8020</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadooptemp/${user.name}</value>
</property>

</configuration>
An example hdfs-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

<property>
<!-- specify this so that running 'hadoop namenode -format' formats the right dir -->
<name>dfs.name.dir</name>
<value>/tmp/hadoop/cache/hadoop/dfs/name</value>
</property>

</configuration>
And an example mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>

<property>
<name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name>
<value>true</value>
</property>

</configuration>



Happy hacking with hadoop!