Creating a generator function for a large data set for a neural network The Next CEO of Stack OverflowSetting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
Why is quantifier elimination desirable for a given theory?
Bartok - Syncopation (1): Meaning of notes in between Grand Staff
Running a General Election and the European Elections together
If Nick Fury and Coulson already knew about aliens (Kree and Skrull) why did they wait until Thor's appearance to start making weapons?
What steps are necessary to read a Modern SSD in Medieval Europe?
Why, when going from special to general relativity, do we just replace partial derivatives with covariant derivatives?
How do I align (1) and (2)?
Example of a Mathematician/Physicist whose Other Publications during their PhD eclipsed their PhD Thesis
Chain wire methods together in Lightning Web Components
Won the lottery - how do I keep the money?
Does increasing your ability score affect your main stat?
Does it make sense to invest money on space investigation?
Flying from Cape Town to England and return to another province
Is micro rebar a better way to reinforce concrete than rebar?
Method for adding error messages to a dictionary given a key
The exact meaning of 'Mom made me a sandwich'
Is it my responsibility to learn a new technology in my own time my employer wants to implement?
How to install OpenCV on Raspbian Stretch?
A Man With a Stainless Steel Endoskeleton (like The Terminator) Fighting Cloaked Aliens Only He Can See
Newlines in BSD sed vs gsed
Which one is the true statement?
I want to delete every two lines after 3rd lines in file contain very large number of lines :
Easy to read palindrome checker
I believe this to be a fraud - hired, then asked to cash check and send cash as Bitcoin
Creating a generator function for a large data set for a neural network
The Next CEO of Stack OverflowSetting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
4
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 2 days ago
m_goldberg
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
4
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 2 days ago
m_goldberg
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
4
4
4
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited 2 days ago
m_goldberg
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
$endgroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
neural-networks
edited 2 days ago
m_goldberg
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
edited 2 days ago
m_goldberg
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
edited 2 days ago
m_goldberg
87.9k872199
edited 2 days ago
m_goldberg
87.9k872199
edited 2 days ago
m_goldberg
87.9k872199
87.9k872199
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
asked 2 days ago
Michel MesedahlMichel Mesedahl
262
262
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
5
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
5
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
add a comment |
5
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
add a comment |
5
5
5
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
$endgroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
answered 2 days ago
Alexey GolyshevAlexey Golyshev
5,0971739
5,0971739
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
add a comment |
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
2 days ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
11 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (
Print@StreamPosition[f] inserted in generator). For your 66 GB file, you must change MaxTrainingRounds (to millions or billions).$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (
Print@StreamPosition[f] inserted in generator). For your 66 GB file, you must change MaxTrainingRounds (to millions or billions).$endgroup$
– Alexey Golyshev
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
6 hours ago
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
