Spark: how to process tree aggregation and statistic2019 Community Moderator ElectionPerformance profiling and tuning in Apache SparkScan-based operations Apache SparkHow to select particular column in Spark(pyspark)?ARIMAX with spark-timeseriesApache Spark QuestionMachine Learning in SparkLoading and querying a Spark machine learning model outside of SparkInstall Spark and Hadoop in the same machineAPI to find out how many executors are running my Spark jobs?Item Similarity with Spark MLLib
What's a natural way to say that someone works somewhere (for a job)?
What will be the benefits of Brexit?
Valid Badminton Score?
Hide Select Output from T-SQL
Bash method for viewing beginning and end of file
If you attempt to grapple an opponent that you are hidden from, do they roll at disadvantage?
How does it work when somebody invests in my business?
Finding all intervals that match predicate in vector
Opposite of a diet
when is out of tune ok?
Why did Kant, Hegel, and Adorno leave some words and phrases in the Greek alphabet?
Do I need a multiple entry visa for a trip UK -> Sweden -> UK?
How can I get through very long and very dry, but also very useful technical documents when learning a new tool?
Can I Retrieve Email Addresses from BCC?
Why is delta-v is the most useful quantity for planning space travel?
Star/Wye electrical connection math symbol
Is the destination of a commercial flight important for the pilot?
What is the term when two people sing in harmony, but they aren't singing the same notes?
apt-get update is failing in debian
Displaying the order of the columns of a table
How do I rename a LINUX host without needing to reboot for the rename to take effect?
Coordinate position not precise
Teaching indefinite integrals that require special-casing
Why Were Madagascar and New Zealand Discovered So Late?
Spark: how to process tree aggregation and statistic
2019 Community Moderator ElectionPerformance profiling and tuning in Apache SparkScan-based operations Apache SparkHow to select particular column in Spark(pyspark)?ARIMAX with spark-timeseriesApache Spark QuestionMachine Learning in SparkLoading and querying a Spark machine learning model outside of SparkInstall Spark and Hadoop in the same machineAPI to find out how many executors are running my Spark jobs?Item Similarity with Spark MLLib
$begingroup$
I have a description of file system in form of a csv:
path;size;inode;type
/folder1/folder1.1;5;1;d
/folder1/folder1.1/file1;10;1;f
/folder1/folder1.1/file2;30;2;f
/folder1/folder1.1/folder1.1.1;5;4;d
/folder1/folder1.1/folder1.1.1/file3;300;5;f
/folder1/folder1.1/folder1.1.1/file4;20;6;f
/folder1/folder1.2;5;7;d
/folder1/file5;30;8;f
/folder1/file6;70;9;f
....
If I put this information in a RRD I will have 4 columns and I could make easy aggregation such as the sum of the size.
However here my goal would be to have a aggregation by folder of different stat such as:
- Recursive file count per folder
- Recursive total size of each folder
- Recursive mean size of file for each folder
If I stop to those 3 values my result could look like this :
path;file count;total size of folder;mean file size in folder
/folder1;6;475;52
/folder1/folder1.1;4;365;73
/folder1/folder1.1/folder1.1.1;2;320;160
/folder1/folder1.2;0;0;0
I have several questions about that kind a treatment :
- Is it relevant to use Spark for such calculation ( I have almost 500 Go of CSV to handle) ?
- If relevant, what would be the best way to approach the problem, should I split my path column and do sub aggregation or should I use method such as
TreeReduceandTreeAggregateor should I use an other way ?
Thanks for any advise on how to handle such problem and feel free to move this question to any Stack site if it belongs elsewhere.
apache-spark
asked Mar 21 at 11:07
KiwyKiwy
1136
$endgroup$
add a comment |
$begingroup$
I have a description of file system in form of a csv:
path;size;inode;type
/folder1/folder1.1;5;1;d
/folder1/folder1.1/file1;10;1;f
/folder1/folder1.1/file2;30;2;f
/folder1/folder1.1/folder1.1.1;5;4;d
/folder1/folder1.1/folder1.1.1/file3;300;5;f
/folder1/folder1.1/folder1.1.1/file4;20;6;f
/folder1/folder1.2;5;7;d
/folder1/file5;30;8;f
/folder1/file6;70;9;f
....
If I put this information in a RRD I will have 4 columns and I could make easy aggregation such as the sum of the size.
However here my goal would be to have a aggregation by folder of different stat such as:
- Recursive file count per folder
- Recursive total size of each folder
- Recursive mean size of file for each folder
If I stop to those 3 values my result could look like this :
path;file count;total size of folder;mean file size in folder
/folder1;6;475;52
/folder1/folder1.1;4;365;73
/folder1/folder1.1/folder1.1.1;2;320;160
/folder1/folder1.2;0;0;0
I have several questions about that kind a treatment :
- Is it relevant to use Spark for such calculation ( I have almost 500 Go of CSV to handle) ?
- If relevant, what would be the best way to approach the problem, should I split my path column and do sub aggregation or should I use method such as
TreeReduceandTreeAggregateor should I use an other way ?
Thanks for any advise on how to handle such problem and feel free to move this question to any Stack site if it belongs elsewhere.
apache-spark
asked Mar 21 at 11:07
KiwyKiwy
1136
$endgroup$
add a comment |
$begingroup$
I have a description of file system in form of a csv:
path;size;inode;type
/folder1/folder1.1;5;1;d
/folder1/folder1.1/file1;10;1;f
/folder1/folder1.1/file2;30;2;f
/folder1/folder1.1/folder1.1.1;5;4;d
/folder1/folder1.1/folder1.1.1/file3;300;5;f
/folder1/folder1.1/folder1.1.1/file4;20;6;f
/folder1/folder1.2;5;7;d
/folder1/file5;30;8;f
/folder1/file6;70;9;f
....
If I put this information in a RRD I will have 4 columns and I could make easy aggregation such as the sum of the size.
However here my goal would be to have a aggregation by folder of different stat such as:
- Recursive file count per folder
- Recursive total size of each folder
- Recursive mean size of file for each folder
If I stop to those 3 values my result could look like this :
path;file count;total size of folder;mean file size in folder
/folder1;6;475;52
/folder1/folder1.1;4;365;73
/folder1/folder1.1/folder1.1.1;2;320;160
/folder1/folder1.2;0;0;0
I have several questions about that kind a treatment :
- Is it relevant to use Spark for such calculation ( I have almost 500 Go of CSV to handle) ?
- If relevant, what would be the best way to approach the problem, should I split my path column and do sub aggregation or should I use method such as
TreeReduceandTreeAggregateor should I use an other way ?
Thanks for any advise on how to handle such problem and feel free to move this question to any Stack site if it belongs elsewhere.
apache-spark
asked Mar 21 at 11:07
KiwyKiwy
1136
$endgroup$
I have a description of file system in form of a csv:
path;size;inode;type
/folder1/folder1.1;5;1;d
/folder1/folder1.1/file1;10;1;f
/folder1/folder1.1/file2;30;2;f
/folder1/folder1.1/folder1.1.1;5;4;d
/folder1/folder1.1/folder1.1.1/file3;300;5;f
/folder1/folder1.1/folder1.1.1/file4;20;6;f
/folder1/folder1.2;5;7;d
/folder1/file5;30;8;f
/folder1/file6;70;9;f
....
If I put this information in a RRD I will have 4 columns and I could make easy aggregation such as the sum of the size.
However here my goal would be to have a aggregation by folder of different stat such as:
- Recursive file count per folder
- Recursive total size of each folder
- Recursive mean size of file for each folder
If I stop to those 3 values my result could look like this :
path;file count;total size of folder;mean file size in folder
/folder1;6;475;52
/folder1/folder1.1;4;365;73
/folder1/folder1.1/folder1.1.1;2;320;160
/folder1/folder1.2;0;0;0
I have several questions about that kind a treatment :
- Is it relevant to use Spark for such calculation ( I have almost 500 Go of CSV to handle) ?
- If relevant, what would be the best way to approach the problem, should I split my path column and do sub aggregation or should I use method such as
TreeReduceandTreeAggregateor should I use an other way ?
Thanks for any advise on how to handle such problem and feel free to move this question to any Stack site if it belongs elsewhere.
apache-spark
apache-spark
asked Mar 21 at 11:07
KiwyKiwy
1136
asked Mar 21 at 11:07
KiwyKiwy
1136
asked Mar 21 at 11:07
KiwyKiwy
1136
asked Mar 21 at 11:07
KiwyKiwy
1136
asked Mar 21 at 11:07
KiwyKiwy
1136
1136
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47731%2fspark-how-to-process-tree-aggregation-and-statistic%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47731%2fspark-how-to-process-tree-aggregation-and-statistic%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
