Dealing with very large dataset (continuation)

-
 Because I have recently updated my operating system (OS), I am wondering how the 64-bit versions of Knime 2.4.2 and RapidMiner 5.1.011 could handle a very large dataset, which cannot be loaded into main memory on a 32-bit OS. This article completes a previous study where we deal with a moderate sized dataset with 500,000 instances and 22 variables. Here, we handle a dataset with 9,634,198 instances and 41 variables. We have already used this dataset in another tutorial. We showed that we cannot perform a decision tree induction on this kind of database without a swapping system, which is implemented into the SIPINA, on a 32-bit OS. We note that Tanagra can handle the dataset, but this is because it encodes the values of the categorical attributes with a byte. The memory occupation remains moderate.

In this tutorial, I analyze the behavior of the 64-bit Knime and RapidMiner on this database. I use 64-bit OS and tools, but I have "only" 4 GB of available memory on my personal computer.

Keywords: very large dataset, decision tree, sampling, sipina, knime, rapidminer
Components: ID3
Tutorial: en_Tanagra_Tree_Very_Large_Dataset.pdf
Dataset: twice-kdd-cup-discretized-descriptors.zip
References:
Tanagra, "Dealing with very large dataset in Sipina".
Tanagra, "Decision tree and large dataset (continuation)".
Tanagra, "Decision tree and large dataset".
Tanagra, "Local sampling for decision tree learning".

Comments

Post a Comment

Popular posts from this blog

Around Here Thirty & Thirty-One 07/21-08/03

-
Image
A glimpse into what it is like to live in our home just this minute.     Intentional Outdoor Hours : 392+ hours (of 1000) Up 44 hours in the last two weeks which is pretty good, but I still need to push myself to grab as many hours as possible in these last few weeks of summer vacation. I got in outdoor time with dog walks, eating dinner outside (my summer fav!), and fires on the patio.  We got to have a long chat at our picnic table with cousin Heather while our kids went bonkers on the trampoline (which had been out of commission for the last month because I ordered the wrong size replacement mat!) Reading Bird by Bird by Anne Lamott and taking it so slow because every time I pick it up it is so thought provoking about the writing process that I need a few days to digest it and think about my own writing. So I also started Sisterland by Curtis Sittenfeld which I snagged off of my sweet friend Shelly and I'm enjoying it.  I continually am shocked each time the writer really show
Read more

connection.

-
One of my favorite quotes credited to Ayesha Siddiqi says, "Be the person you needed when you were young." When I was young, I needed someone who had a different perspective - a wider lens of life and humanity and culture and experiences than our small (albeit beloved to me) town. I wanted to see more and learn more and experience more. And I was lucky enough that my parents and teachers could see this in me that they pushed me towards those people. My parents (with reluctance but good faith) put me on an airplane headed to Honduras with a group of doctors and dentists when I was sixteen. They also became what I needed themselves, my Dad the curious researcher and questioner - my Mom the ever patient listening ear while I went on and on about some new thing I learned. As a teacher now, I see everyday that what so many of my students need is someone who simply has the time of day to notice them. I have students everyday starved for meaningful connection. The person our young p
Read more

Data Mining with R - The Rattle Package

-
R ( http://www.r-project.org/ ) is one of the most exciting free data mining software projects of these last years. Its popularity is absolutely justified (see Kdnuggets Polls - Data Mining/ Analytic Tools Used - 2011 ). Among the reasons which explain this success, we distinguish two very interesting characteristics: (1) we can extend almost indefinitely the features of the tool with the packages; (2) we have a programming language which allows to perform easily sequences of complex operations. But this second property can be also a drawback. Indeed, some users do not want to learn a new programming language before being able to realize projects. For this reason, tools which allow to define the sequence of commands with diagrams (such as Tanagra, Knime, RapidMiner, etc.) still remain a valuable alternative with the data miners. I n this tutorial, we present the "Rattle" package which allows to the data miners to use R without needing to know the associated programming langua
Read more