Your Laptop Can Now Analyze Big Data

New software makes it possible to do in minutes on a small computer what used to be done by large clusters of computers.

Computer scientists from Carnegie Mellon University have devised a framework for running large-scale computations for tasks such as social network or Web search analysis efficiently on a single personal computer.

The software could help developers working on many modern tasks: for example, designing a new recommendation engine using social network connections. In order to make effective recommendations—"your friends liked this movie, so here is another movie that you haven't seen yet, but you will probably like"—the software has to be able to analyze the connections between the members of a social network. This type of task is called graph computation, and it is increasingly common. But working with large-scale data sets (such as online social networks) usually requires the processing horsepower of many computers clustered together, such as those offered by Amazon's cloud-based EC2 service.

The new software, called GraphChi, exploits the capacious hard drives that are becoming ever more common in personal computers. A graph would normally be stored in temporary memory (RAM) for analysis. With GraphChi, the hard drive performs this task instead.

"PCs don't have enough RAM to hold an entire Web graph, but they do have hard drives, which can hold a lot of information," says Carlos Guestrin, codirector of Carnegie Mellon's Select Lab, where GraphChi was developed. But hard drives are slow compared to RAM for reading and writing data, which tends to slow down computation. So Guestrin's student AapoKyrola designed a faster, less random method of accessing the hard drive.

According to Guestrin, a Mac Mini running GraphChi can analyze Twitter's social graph from 2010—which contains 40 million users and 1.2 billion connections—in 59 minutes. "The previous published result on this problem took 400 minutes using a cluster of about 1,000 computers," Guestrin says.

As technology gets more networked, and data sets get larger, graph computation is becoming more and more relevant in many domains, says David A. Bader, a graph computation expert at Georgia Tech. "Trying to understand how the human brain works or trying to make sense of medical patient records involve graph computing," he says.

Graph analysis also drives the development of new web products, says Jeremy Kepner, a researcher at MIT. "Document search, ad placement, route planning, travel reservations, and cyber security all rely on graph analysis," he says. "Enabling web developers to construct these analyses on their desktop computers catalyzes these industries and accelerates product development."

Guestrin adds that GraphChi can handle "streaming graphs," which more accurately model large networks by showing how relationships change over time. Bader and others at Georgia Tech have created a graph computation framework, called Stinger, that's optimized for supercomputers working with massive streaming graphs.

"The scales of these problems will obviously keep growing," says Guestrin. But he says GraphChi is capable of effectively handling many large-scale graph-computing problems without resorting to cloud-based solutions or supercomputers.

"A researcher in computational biology could do large-scale computations on their PC; a developer working on a data-center algorithm can test it on their laptop before pushing it to the cloud," Guestrin says. "Big data is everywhere now, but some big data isn't as big as it once was, relatively speaking. Tools like GraphChi will let many companies and startups solve all their graph-computing needs on a single machine. It's cost effective, and it drives innovation, too."

Cybercriminals in developing nations targeted

Cybercriminals in developing nations are being targeted in a new effort to combat the illegal activity.

The International Cyber Security Protection Alliance has launched a research project to identify how attacks are likely to evolve over the next eight years.

It said that faster links to the net in parts of Africa, Asia and Eastern Europe were likely to create problems.

The ICSPA will advise governments and local authorities on how best to react.

The organisation is a coalition of law agencies, security companies and businesses, including Visa Europe and the defence firm EADS.

"It's not just about putting a black mark against a particular nation because many of these countries are the unwilling hosts to cybercriminal networks," John Lyons, the organisation's chief executive, told the BBC.

"We know the countries that provide 'organised cybercrime' with the ability and the hosting capability to attack the West in terms of its business and customers.

"So, what the ICSPA is looking to do is to work with those nations to provide support to help them improve the cyber-resilience of their national infrastructure, to aid their own economies, and to help their law enforcement groups tackle cybercriminals who work out of their country."

Advising Africa

Although the research project has only just got underway, the group has already started to co-ordinate action.

Members of the UK's Serious Organised Crime Agency (Soca) began work with the Ghanaian government in February to help it take technical steps to prevent cybercrime, and ensure offenders are prosecuted.

Mr Lyons said Nigeria, Botswana, Uganda and Rwanda were among nations likely to be targeted by the campaign.

The countries were starting to benefit from improved internet access, but would struggle to "suppress the criminality that will come with that connectivity," he said.

Other countries on his group's watch list included Bulgaria, Romania, India, the Philippines and parts of Latin America.

October report

The ICSPA wants companies based in developing nations to contribute to the costs of its efforts.

However, Mr Lyons acknowledged that the richer nations also needed to do more to combat internet crime.

"Something like 67% of malware which is used to attack Western businesses is hosted in the US on servers," he said. "The US needs to take steps to tackle that particular issue."

The ICSPA plans to issue a report covering its initial findings before the end of October.

Nigeria's low-cost tablet computer


Nigeria's Saheed Adepoju is a young man with big dreams. He is the inventor of the Inye, a tablet computer designed for the African market.

According to the 29-year-old entrepreneur, his machine's key selling point is its price - $350 (£225) opposed to around $700 for an iPad.

He believes that, because of this, there is a big market for it in Nigeria and elsewhere in Africa, particularly amongst students.

He is also hoping to sell his tablet - which runs on the Google Android operating system - to the Nigerian government and plans to have at least one computer in each local government area

"The Inye is a mobile internet device. It gives you access to the internet; it allows you to play media files and watch movies. What we have is an 8-inch device, a device that is half-way between a laptop and a mobile phone," he told the BBC's series African Dream.

"You have the standard software applications that come pre-installed and then you have the ones that we are working with various local developers to bundle on," he added.

Among those local apps there is one designed to raise awareness about HIV and others related to water and sanitation.

"We work with local developers that have expertise in particular areas so that we don't end up doing so much work and we just have a collaborative way of doing things together," he said