Gain More Knowledge...

Hadoop 1.0.0 single node configuration on ubuntu

2012-01-27T17:55:00.014+05:30

Hadoop is a framework for distributed processing across multiple compute clusters. It provides reliable data storage using Hadoop Distributed File System(HDFS) and high performance parallel data processing using MapReduce method. You can find more information from following

http://wiki.apache.org/hadoop/

Here I am describing my own experience with hadoop 1.0.0 configuration in a ubuntu box. I am using ubuntu 11.04 for this configuration.

Step 1: Download and install oracle jdk

Install jdk 1.6 or above using following step
Add the Canonical Partner Repository to your apt repositories:

hadooptest@hadooptest-VM$ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"

Update the source list

hadooptest@hadooptest-VM$ sudo apt-get update

Install sun-java6-jdk

hadooptest@hadooptest-VM$ sudo apt-get install sun-java6-jdk

Select Sun’s Java as the default on your machine.

hadooptest@hadooptest-VM$ sudo update-java-alternatives -s java-6-sun

After the installation check the java version using

hadooptest@hadooptest-VM$java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

Step 2:Download and Install Hadoop

Download i386 or amd64 version(according to your os version) of .deb package from http://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-1.0.0/. Install the hadoop by double clicking the file or using dpkg command.

hadooptest@hadooptest-VM$sudo dpkg -i  hadoop_1.0.0-1_i386.deb

Step 3: Set up Hadoop for single node

Setup hadoop for single node using following command

hadooptest@hadooptest-VM$sudo hadoop-setup-single-node.sh

Answer "yes" for all questions. Service will automatically started after the installation.

Step 4: Test hadoop configuration

hadooptest@hadooptest-VM$ sudo hadoop-validate-setup.sh --user=hdfs

If you get "teragen, terasort, teravalidate passed." near the end of the output, everything is ok.

Hadoop Tracker websites

JobTracker website: http://localhost:50030/
NameNode website : http://localhost:50070/
Task track website: http://localhost:50060/

Step 5: Example MapReduce job using word count

5.1 Download Plain Text UTF-8 encoding file for following books and store into a local directory (here using /home/hadooptest/gutenberg)

5.2 Download mapreduce programme jar(hadoop-examples-0.20.203.0.jar) file to any local folder (here using /home/hadooptest)
5.3. To run mapreduce programe, we need to copy these files into HDFS directory from local directory. For this purpose, first login to the hadoop user using

hadooptest@hadooptest-VM$su hdfs

Copy local file to HDFS using

hdfs@hadooptest-VM$hadoop dfs -copyFromLocal /home/hadooptest/gutenberg /user/hdfs/gutenberg

Check the content inside HDFS directory using

hdfs@hadooptest-VM$hadoop dfs -ls /user/hdfs/gutenberg

5.4. Move to folder that containing downloaded jar file.
5.5. Run the following command to execute the programme

hdfs@hadooptest-VM:/home/hadooptest$hadoop jar /user/hdfs/hadoop-examples-0.20.203.0.jar /user/hdfs/gutenberg /user/hdfs/gutenberg-out

Here /user/hdfs/gutenberg is the input directory and /user/hdfs/gutenberg-out is the output directory. Both input and output directory must be in HDFS file system.
It will take some time according to your system configuration. You can track the job progress using hadoop tracker websites

5.6. Check the result of the programme using

hdfs@hadooptest-VM:/home/hadooptest$hadoop dfs -cat /user/hduser/gutenberg-output/part-r-00000

Install Xen 4.1 on Ubuntu 11.10

2011-12-30T17:48:00.015+05:30

Ubuntu support xen officially from 11.10 version. But there is some issue while creating DomU. Here find some easy step to configure xen 4.1 on 64 bit version of ubuntu 11.10 and solve issue in DomU creation.

1. Install xen and utilities

$ sudo apt-get install xen-hypervisor-4.1-amd64 xen-utils-4.1 xenwatch xen-tools xen-utils-common xenstore-utils
$sudo apt-get install virtinst
$sudo apt-get install virt-viewer virt-manager

2.Restart OS and Select Xen Kernal

Verify the Xen installation using

$sudo xm info

If this command does not return any error, then the installation is correct.

3.Xend Configuration

Edit /etc/xen/xend-config.sxp and uncomment this line

(xend-unix-server no)

and change to

(xend-unix-server yes)

Edit .bashrc file using

#vi ~/.bashrc , add the following line:

export VIRSH_DEFAULT_CONNECT_URI="xen:///"

4. Restart OS and Select Xen Kernel

Verify libvirt Installation

$sudo virsh version

Compiled against library: libvir 0.8.3
Using library: libvir 0.8.3
Using API: Xen 3.0.1
Running hypervisor: Xen 4.0.0

If got output like this, you have installed every package correctly.

5.Creating VM using virt-manager

Here I am telling about virtual machine creation using virt-manger

Type $sudo virt-manager for getting GUI for virt-manager

Create new VM using virt-manager(Learn More)

Solution for common error during DomU is given below..

1. Show Something like this

Fix it using

$sudo mkdir /usr/lib64/xen -p
$sudo cp /usr/lib/xen-4.1/* -r /usr/lib64/xen/

2. Get like this

Solve it using

$sudo mkdir /usr/share/qemu

$sudo cp -r /usr/share/qemu-linaro/keymaps /usr/share/qemu/

How To Disable Guest Session in Ubuntu 11.10 Oneiric Ocelot

2011-11-20T14:30:00.005+05:30

Ubuntu 11.10 added guest account by default.This is the part of new display manager lightDM. To disable his feature follow the below step

1. Open terminal
2. Type ' sudo vi /etc/lightdm/lightdm.conf ' .Then you got a file like this



[SeatDefault]
greeter-session=unity-greeter
user-session=ubuntu

3. Add the line 'allow-guest=false' to the end of the file
4. Save file and exist
5. Reboot your system or restart lightdm service using “sudo restart lightdm” command

For video demonstration visit: http://www.youtube.com/watch?v=qBmF6rZCYh8

Mount Virtual Box Image in Ubuntu

2011-07-24T11:24:00.008+05:30

VirtualBox provides a tool called vdfuse, which is a FUSE-based filesystem package that can mount any VirtualBox VDI image.It support VDI, VMDK, VHD and raw format

Install vdfuse package using the following command

$ sudo apt-get install virtualbox-ose-fuse

You can now mount the VDI file like this:

$ sudo vdfuse [options] -f /path/to/file.vdi /path/to/mountpoint

The options are

-h help
-r readonly
-t specify type (VDI, VMDK, VHD, or raw; default: auto)
-f VDimage file
-a allow all users to read disk
-w allow all users to read and write to disk
-g run in foreground
-v verbose
-d debug

This command creates file like 'EntireDisk', 'Partition1', etc at the mount point.

To mount the filesystem, just use:

mount /path/to/mountpoint/Partition1 /path/to/someother/mountpoint

Then view the file system at /path/to/someother/mountpoint

Whitepaper on Getting Started With Eucalyptus Cloud

2011-04-19T15:23:00.010+05:30

Abstract

Cloud computing is a new area of computation systems which provide users with access to large amounts of computer resources and data. The principle behind the cloud is that any computer connected to the internet is connected to the same pool of computing power, applications, and files.Eucalyptus is an open source cloud computing framework provide Infrastructure as a Service(IaaS). It can be used to create private and public cloud.It provides an EC2-compatible cloud computing platform and S3-compatible cloud storage platform.
Download full Pdf

Error

1. While installing os to image file, system crash by showing the error "there is no enough space on hard disk" but already system has 10GB memory
2. While accessing eucalyptus community cloud, port is not opened

Whitepaper on Misuse and Anomaly Based Model For Botnet Detection

2011-03-16T13:46:00.002+05:30

Abstract

Today bot and botnet become the primary platform for various types of attacks in the internet. Because of this reason, there are many types of detection mechanisms that are proposed to detect the bot activity. These detection mechanisms are based on common malware detection model. Most of the existing systems uses two types of models – misuse and anomaly based. In misuse detection model, the system detects the attacks based on known signatures where as, in anomaly detection model it is based on abnormal behaviour of the system. Both these methods have its own advantages and disadvantages.

Download pdf

Kollam cyber cell visit

2011-01-08T12:54:00.013+05:30

Date: 29 December 2010

Team Members: Jinesh M.K, Manu R Krishnan, Vinitha V Prabhu, Shyju Mol

• What is cybercrime?

Any crime which is done using devices which have an IC (Integrated Chip) is termed as a cybercrime. So comes the doubt that is taking a photo in a mobile phone is a crime or not? According to cyber law it is not actually a crime, but capturing, editing and transferring is a crime

• Procedures to complaint a cyber crime

Initially we have to give written complaint to our nearest local police station,but not directly to cyber cell. If the complaint comes under cyber law, then it will be forward to cyber cell.
The main aim of cyber cell is to just assist the investigation team having technical support.

• Main area of crime they handle?

Most of the crime they handle is based on mobile phones and computer. But because of infrastructure limitations in the district cyber cell, they will handle only crime related to mobile phones.

• How do victims can help in the investigation?

“Crime may be any type,but only with victim’s help , investigation can be speed up”.
If we take an example, cybercrime related to mobile phone.ie when we lose our mobile phone, IMEI(type *#06#) number of our mobile phone will help cyber cell team to find the phone easily.
So it will be better to save IMEI number of your mobile in secure place (not in your mobile phone)

• Cyber cell team in the crime scene

In the case of computer system
1. If it’s working, pull down CPU power cable(Why? post your comment)
2. Seize and seal all the connected devices along with that system in the presences of a witness having computer knowledge
3. Send all this devices to responsible authority for analysis

• Who are these authorities?

C-DAC and cyber forensics lab.

• How to find the accused?

Let us take an example: suppose a person’s account get hacked and post bad comments, or send fake mails ,then according to victim’s complaint the cyber team will contact with corresponding service providers and get login details, ip address and location of machine. If it is an internet café, then with the register team will find the accused and if it is a home then with the house owner team can find who were the visitors of his computer.

• What will do if accused is giving fake address?

Actually the cyber team states that the internet café staffs has to check the identity of each person entering to the café. Now a day’s no one is following this rule .But if the location to find the accused is a internet café then the owner of café will be arrested along with the accused.

• What will do if accused is giving spoofed ip (fake ip)?

In all our mail servers like yahoo, gmail, there is an option called full header option. when we go through that option we can find the path through which the mail has arrived.

Speedy Chrome 7>>>>60 times faster

2010-09-22T13:28:00.003+05:30

Google announce Google chrome version 7 that achieving speeds 60 times faster than in version 6.

This system picks the best graphics API to use on each OS that Chromium supports: Windows XP/Vista/7, Mac OS and Linux. The latest beta version of Mozilla’s Firefox 4 uses hardware acceleration for faster speeds. Google also use this feature for achieving the speed.It have following feature

1. It won’t crash: Because of multiprocessing architecture

2. It’s really fast: Using hardware acceleration

3. More space for browsing: Screen size is devoted to the site with no buttons and logos

4. It makes searching simpler.

5. It gives you more control over tabs.

You can grab a tab and drag it out into its own individual window. Or you can drag and drop tabs into existing windows to combine them.

6. It opens new doors on your home page.

Chrome comes with a default dynamic home page. As you use it, the program remembers the sites that you visit most often.

Also read this article from www.amritaayanam.com

Beta version download click here

Search Engine Privacy

2010-09-08T11:00:00.001+05:30

Today,Google is the mostly used search engine.According to new survey 70% people using Google for their web search. But do you know, when you access Google services,servers automatically record information that your browser sends whenever you visit a website. These server logs may include information such as your web request, Internet Protocol address, browser type, browser language, the date and time of your request

In this context privacy of searching become an issue. But some of search engine provide great privacy for our searching for example Ixquick (www.ixquick.com) . It focuses on delivering great search results with the best possible privacy. Ixquick is known as Startpage in the United States (www.startpage.com).Ixquick has the industry's leading Privacy Policy: No recording of users' IP addresses. No identifying cookies. No collection of personal data. No sharing personal data with third parties. Offers secure encrypted connections (HTTPS/SSL). And a free proxy service that allows anonymous browsing of websites.

Fake 'Dislike' Button Scam Breaks Out On Facebook

2010-08-17T18:42:00.000+05:30

IT security and control firm Sophos is advising Facebook users to be wary of the latest survey scam spreading virally across the network. There are a number of variations of this scam, which sees users unwillingly update their Facebook status encouraging others to get the 'official Dislike button'. The scam is spreading quickly as many

Facebook users have been calling for the introduction of an official 'Dislike' feature which would allow them to express their opinions on other users' posts, links and updates.

How to prevent trojan

2010-08-16T12:40:00.001+05:30

Make sure your anti-virus software is up to date.
Keep firewalls set to the highest level.
Never open an e-mail attachment from someone you don't know.
Never double-click on an e-mail attachment that ends in .exe. It is an 'executable' file and can do what it likes in your system.
If you think your machine has already been infected, contact your bank immediately. If the bank thinks you are a genuine victim of fraud it will reimburse you.

'most dangerous trojan '

2010-08-16T12:32:00.003+05:30

Cyber criminals have raided the accounts of thousands of British internet bank customers in one of the most sophisticated attacks of its kind. Attack effect around 3,000 people.
The attacks were carried out using trojan horse.The latest attack involved a Trojan called Zeus v3 which hides inside adverts on legitimate website.
British security team estimate that at least £675,000 has been illegally transferred from the UK in the last month - and that the attacks are still continuing.
In this context we must aware about the risk in internet banking and take necessary step to prevent this......

2611:A National Emergency Telephone Number

2010-08-10T13:43:00.000+05:30

A National Emergency Telephone Number - 2611 - has been mooted by a Pune-based Telecom engineer as a versatile communication platform during emergency and disaster management in the country. Dial 2611, a National Emergency Telephone Number system (NETNS,) is designed and developed as a nationwide single telephone number assigned to integrate local, regional and national help lines and is easy to remember

Mozilla available in Malayalam

2010-08-04T09:06:00.003+05:30

Download the new Version of Mozilla Firefox in Malayalam. Enjoy the internet in your language
for download click here

Intel Demos Light Over Silicon

2010-07-30T16:22:00.001+05:30

Microprocessor maker Intel (Nasdaq: INTC) has announced that it has developed a research prototype that it says is the world's first silicon-based optical data connection with integrated lasers. The link can move up to 50 gigabits of data per second -- about as much data as that contained in an entire HD movie.

The shortage of IPv4 addresses has reached a critical stage

2010-07-27T12:17:00.000+05:30

The Number Resource Organization (NRO), which represents the registries, said that less than 10 percent of all IPv4 addresses remain available, threatening the future network operations of all businesses and organizations unless ISPs and businesses step up their migration to IPv6.The remaining addresses available up to 12 months. Therefore it is the time for migration to IPv6

iPhone 4 in India

2010-07-21T14:11:00.000+05:30

Bharti Airtel has announced that it will bring iPhone 4 in India by October.

Features

Full stainless steel body with glass front
Proximity sensor for dimming control and auto answer
24% thinner than previous 3GS
New ‘retinal display’
960×640 resolution
New battery
Quad-band radio
3-axis gyroscope
Proximity sensor
5 megapixel rear camera
VGA front-facing camera
720 HD recording
iMovie for iPhone

Microsoft Outlook Connects to Facebook

2010-07-19T18:33:00.000+05:30

Microsoft's enabled you to access Facebook in your email inbox. They've created an 'application' in Outlook called Outlook Social Connector which links to your Facebook account, syncing your contacts to your friends' Facebook profiles.

Font with indian rupee symbol

2010-07-17T16:41:00.000+05:30

First font that support Indian rupee symbol

Select Rupee font and press ~(above tab) for this symbol
Freedownload

The first web browser for India

2010-07-17T12:33:00.000+05:30

The first-ever web browser for India released, called "Epic Browser" . Epic Browser is a web browser which is based on Mozilla Firefox. It services are useful for Indian users.

For free download click here

nesc Plugin for Eclipse

2010-07-15T12:27:00.000+05:30

It is very useful for nesc Programming

Do the folllowing step

1.Download the nesdt.jar file[for download press here]
2.copy this file into eclipse plugin folder
3.Start new nesc project and link with tiny os folder

New Open Source OS

2010-07-12T09:48:00.000+05:30

New open source OS will feature 'disposable' virtual machines

A new open-source operating system will come with the option of creating one-time, disposable virtual machines on the fly as a way to protect against malicious files.

Read More>>

4k-resolution videos on YouTube

2010-07-10T09:16:00.000+05:30

Famous videos sharing site, You-Tube now support 4k.4k is an industry-standard format that makes 1080p-resolution clips.

Speed up your laptop

2010-07-09T15:18:00.000+05:30

Turn on the classic mode
Turn off Visual effects in System
Turn off System Restore in System
Don’t use any background wallpaper
Defragment your hard disks regularly

tiny os plugin

2010-07-09T14:12:00.000+05:30

Yeti 2 - TinyOS 2 Plugin for Eclipse

Try it,easy way for nesC program

http://www.eclipseplugincentral.com/Web_Links-index-req-viewlink-cid-1477.html