Can a neural network compute $y = x^2$? The Next CEO of Stack Overflow2019 Community Moderator ElectionDebugging Neural Network for (Natural Language) TaggingIs ML a good solution for identifying what the user wants to do from a sentence?Which functions neural net can't approximateQ Learning Neural network for tic tac toe Input implementation problemError in Neural NetworkWhat database should I use?Reinforcement learning - How to deal with varying number of actions which do number approximationMultiple-input multiple-output CNN with custom loss functionWhy are neuron activations stored as a column vector?Learning a highly non-linear function with a small data set
Multi tool use
How to start emacs in "nothing" mode (`fundamental-mode`)
What can we do to stop prior company from asking us questions?
LWC - Unit Testing NavigationMixin.GenerateUrl
How easy is it to start Magic from scratch?
Can the Reverse Gravity spell affect the Meteor Swarm spell?
What flight has the highest ratio of time difference to flight time?
Why does the UK parliament need a vote on the political declaration?
Rotate a column
Extracting names from filename in bash
What was the first Unix version to run on a microcomputer?
How should I support this large drywall patch?
Indicator light circuit
Customer Requests (Sometimes) Drive Me Bonkers!
Is there a way to save my career from absolute disaster?
Can a caster that cast Polymorph on themselves stop concentrating at any point even if their Int is low?
Clustering points and summing up attributes per cluster in QGIS
Is it safe to use c_str() on a temporary string?
Beyond letters and diaries - exercises to explore characters' personalities and motivation
Why is the US ranked as #45 in Press Freedom ratings, despite its extremely permissive free speech laws?
Increase performance creating Mandelbrot set in python
At which OSI layer a user-generated data resides?
Should I tutor a student who I know has cheated on their homework?
Why didn't Khan get resurrected in the Genesis Explosion?
How do scammers retract money, while you can’t?
Can a neural network compute $y = x^2$?
The Next CEO of Stack Overflow2019 Community Moderator ElectionDebugging Neural Network for (Natural Language) TaggingIs ML a good solution for identifying what the user wants to do from a sentence?Which functions neural net can't approximateQ Learning Neural network for tic tac toe Input implementation problemError in Neural NetworkWhat database should I use?Reinforcement learning - How to deal with varying number of actions which do number approximationMultiple-input multiple-output CNN with custom loss functionWhy are neuron activations stored as a column vector?Learning a highly non-linear function with a small data set
$begingroup$
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a number (e.g. as a vector in binary form, so that number 5
is represented as [1,0,1,0,0,0,0,...]
), the neural network should learn to return its square - 25 in this case.
If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.
Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.
Can you suggest any ideas or reading on subject?
machine-learning neural-network
$endgroup$
add a comment |
$begingroup$
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a number (e.g. as a vector in binary form, so that number 5
is represented as [1,0,1,0,0,0,0,...]
), the neural network should learn to return its square - 25 in this case.
If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.
Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.
Can you suggest any ideas or reading on subject?
machine-learning neural-network
$endgroup$
add a comment |
$begingroup$
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a number (e.g. as a vector in binary form, so that number 5
is represented as [1,0,1,0,0,0,0,...]
), the neural network should learn to return its square - 25 in this case.
If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.
Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.
Can you suggest any ideas or reading on subject?
machine-learning neural-network
$endgroup$
In spirit of the famous Tensorflow Fizz Buzz joke and XOr problem I started to think, if it's possible to design a neural network that implements $y = x^2$ function?
Given some representation of a number (e.g. as a vector in binary form, so that number 5
is represented as [1,0,1,0,0,0,0,...]
), the neural network should learn to return its square - 25 in this case.
If I could implement $y=x^2$, I could probably implement $y=x^3$ and generally any polynomial of x, and then with Taylor series I could approximate $y=sin(x)$, which would solve the Fizz Buzz problem - a neural network that can find remainder of the division.
Clearly, just the linear part of NNs won't be able to perform this task, so if we could do the multiplication, it would be happening thanks to activation function.
Can you suggest any ideas or reading on subject?
machine-learning neural-network
machine-learning neural-network
edited Mar 22 at 17:25
Boris Burkov
asked Mar 22 at 13:02
Boris BurkovBoris Burkov
1385
1385
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
Neural networks are also called as the universal function approximation which is based in the universal function approximation theorem. It states that :
In the mathematical theory of artificial neural networks,
the universal approximation theorem states that a feed-forward network
with a single hidden layer containing a finite number of neurons can
approximate continuous functions on compact subsets of Rn, under mild
assumptions on the activation function
Meaning a ANN with a non linear activation function could map the function which relates the input with the output. The function y = x^2 could be easily approximated using regression ANN.
You can find an excellent lesson here with a notebook example.
Also, because of such ability ANN could map complex relationships for example between an image and its labels.
$endgroup$
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
add a comment |
$begingroup$
I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $mathbbR^n$, under mild assumptions on the activation function.
But the main problem is that the theorem has a very important
limitation. The function needs to be defined on compact subsets of
$mathbbR^n$ (compact subset = bounded + closed subset). But why
is this problematic?. When training the function approximator you
will always have a finite data set. Hence, you will approximate the
function inside a compact subset of $mathbbR^n$. But we can always
find a point $x$ for which the approximation will probably fail. That
being said. If you only want to approximate $f(x)=x^2$ on a compact
subset of $mathbbR$ then we can answer your question with yes.
But if you want to approximate $f(x)=x^2$ for all $xin mathbbR$
then the answer is no (I exclude the trivial case in which you use
a quadratic activation function).
Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $xto 10000$ if you don't use enough terms in your Taylor expansion.
New contributor
$endgroup$
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47787%2fcan-a-neural-network-compute-y-x2%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Neural networks are also called as the universal function approximation which is based in the universal function approximation theorem. It states that :
In the mathematical theory of artificial neural networks,
the universal approximation theorem states that a feed-forward network
with a single hidden layer containing a finite number of neurons can
approximate continuous functions on compact subsets of Rn, under mild
assumptions on the activation function
Meaning a ANN with a non linear activation function could map the function which relates the input with the output. The function y = x^2 could be easily approximated using regression ANN.
You can find an excellent lesson here with a notebook example.
Also, because of such ability ANN could map complex relationships for example between an image and its labels.
$endgroup$
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
add a comment |
$begingroup$
Neural networks are also called as the universal function approximation which is based in the universal function approximation theorem. It states that :
In the mathematical theory of artificial neural networks,
the universal approximation theorem states that a feed-forward network
with a single hidden layer containing a finite number of neurons can
approximate continuous functions on compact subsets of Rn, under mild
assumptions on the activation function
Meaning a ANN with a non linear activation function could map the function which relates the input with the output. The function y = x^2 could be easily approximated using regression ANN.
You can find an excellent lesson here with a notebook example.
Also, because of such ability ANN could map complex relationships for example between an image and its labels.
$endgroup$
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
add a comment |
$begingroup$
Neural networks are also called as the universal function approximation which is based in the universal function approximation theorem. It states that :
In the mathematical theory of artificial neural networks,
the universal approximation theorem states that a feed-forward network
with a single hidden layer containing a finite number of neurons can
approximate continuous functions on compact subsets of Rn, under mild
assumptions on the activation function
Meaning a ANN with a non linear activation function could map the function which relates the input with the output. The function y = x^2 could be easily approximated using regression ANN.
You can find an excellent lesson here with a notebook example.
Also, because of such ability ANN could map complex relationships for example between an image and its labels.
$endgroup$
Neural networks are also called as the universal function approximation which is based in the universal function approximation theorem. It states that :
In the mathematical theory of artificial neural networks,
the universal approximation theorem states that a feed-forward network
with a single hidden layer containing a finite number of neurons can
approximate continuous functions on compact subsets of Rn, under mild
assumptions on the activation function
Meaning a ANN with a non linear activation function could map the function which relates the input with the output. The function y = x^2 could be easily approximated using regression ANN.
You can find an excellent lesson here with a notebook example.
Also, because of such ability ANN could map complex relationships for example between an image and its labels.
answered Mar 22 at 13:20
Shubham PanchalShubham Panchal
36118
36118
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
add a comment |
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
2
2
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
$begingroup$
Thank you very much, this is exactly what I was asking for!
$endgroup$
– Boris Burkov
Mar 22 at 13:23
2
2
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
$begingroup$
Although true, it a very bad idea to learn that. I fail to see where any generalization power would arise from. NN shine when there's something to generalize. Like CNN for vision that capture patterns, or RNN that can capture trends.
$endgroup$
– Jeffrey
Mar 22 at 15:21
add a comment |
$begingroup$
I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $mathbbR^n$, under mild assumptions on the activation function.
But the main problem is that the theorem has a very important
limitation. The function needs to be defined on compact subsets of
$mathbbR^n$ (compact subset = bounded + closed subset). But why
is this problematic?. When training the function approximator you
will always have a finite data set. Hence, you will approximate the
function inside a compact subset of $mathbbR^n$. But we can always
find a point $x$ for which the approximation will probably fail. That
being said. If you only want to approximate $f(x)=x^2$ on a compact
subset of $mathbbR$ then we can answer your question with yes.
But if you want to approximate $f(x)=x^2$ for all $xin mathbbR$
then the answer is no (I exclude the trivial case in which you use
a quadratic activation function).
Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $xto 10000$ if you don't use enough terms in your Taylor expansion.
New contributor
$endgroup$
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
add a comment |
$begingroup$
I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $mathbbR^n$, under mild assumptions on the activation function.
But the main problem is that the theorem has a very important
limitation. The function needs to be defined on compact subsets of
$mathbbR^n$ (compact subset = bounded + closed subset). But why
is this problematic?. When training the function approximator you
will always have a finite data set. Hence, you will approximate the
function inside a compact subset of $mathbbR^n$. But we can always
find a point $x$ for which the approximation will probably fail. That
being said. If you only want to approximate $f(x)=x^2$ on a compact
subset of $mathbbR$ then we can answer your question with yes.
But if you want to approximate $f(x)=x^2$ for all $xin mathbbR$
then the answer is no (I exclude the trivial case in which you use
a quadratic activation function).
Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $xto 10000$ if you don't use enough terms in your Taylor expansion.
New contributor
$endgroup$
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
add a comment |
$begingroup$
I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $mathbbR^n$, under mild assumptions on the activation function.
But the main problem is that the theorem has a very important
limitation. The function needs to be defined on compact subsets of
$mathbbR^n$ (compact subset = bounded + closed subset). But why
is this problematic?. When training the function approximator you
will always have a finite data set. Hence, you will approximate the
function inside a compact subset of $mathbbR^n$. But we can always
find a point $x$ for which the approximation will probably fail. That
being said. If you only want to approximate $f(x)=x^2$ on a compact
subset of $mathbbR$ then we can answer your question with yes.
But if you want to approximate $f(x)=x^2$ for all $xin mathbbR$
then the answer is no (I exclude the trivial case in which you use
a quadratic activation function).
Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $xto 10000$ if you don't use enough terms in your Taylor expansion.
New contributor
$endgroup$
I think the answer of @ShubhamPanchal is a little bit misleading. Yes, it is true that by Cybenko's universal approximation theorem we can approximate $f(x)=x^2$ with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $mathbbR^n$, under mild assumptions on the activation function.
But the main problem is that the theorem has a very important
limitation. The function needs to be defined on compact subsets of
$mathbbR^n$ (compact subset = bounded + closed subset). But why
is this problematic?. When training the function approximator you
will always have a finite data set. Hence, you will approximate the
function inside a compact subset of $mathbbR^n$. But we can always
find a point $x$ for which the approximation will probably fail. That
being said. If you only want to approximate $f(x)=x^2$ on a compact
subset of $mathbbR$ then we can answer your question with yes.
But if you want to approximate $f(x)=x^2$ for all $xin mathbbR$
then the answer is no (I exclude the trivial case in which you use
a quadratic activation function).
Side remark on Taylor approximation: You always have to keep in mind that a Taylor approximation is only a local approximation. If you only want to approximate a function in a predefined region then you should be able to use Taylor series. But approximating $sin(x)$ by the Taylor series evaluated at $x=0$ will give you horrible results for $xto 10000$ if you don't use enough terms in your Taylor expansion.
New contributor
edited Mar 22 at 17:09
New contributor
answered Mar 22 at 17:03
MachineLearnerMachineLearner
36910
36910
New contributor
New contributor
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
add a comment |
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
2
2
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
$begingroup$
Nice catch! "compact set".
$endgroup$
– Esmailian
Mar 22 at 17:14
1
1
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
$begingroup$
Many thanks, mate! Eye-opener!
$endgroup$
– Boris Burkov
Mar 22 at 17:23
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47787%2fcan-a-neural-network-compute-y-x2%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
JEmewO188Y4Aq6