How to handle columns with categorical data and many unique values The 2019 Stack Overflow Developer Survey Results Are Indecision trees on mix of categorical and real value parametersPandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)Imputation of missing values and dealing with categorical valuesHow to deal with categorical variablesOne hot encoding error “sort.list(y)…”One hot encoding vs Word embeddingHow to implement feature selection for categorical variables (especially with many categories)?ML Models: How to handle categorical feature with over 1000 unique values“Binary Encoding” in “Decision Tree” / “Random Forest” AlgorithmsDealing with multiple distinct-value categorical variables

Button changing its text & action. Good or terrible?

How to support a colleague who finds meetings extremely tiring?

Is it safe to harvest rainwater that fell on solar panels?

How can I add encounters in the Lost Mine of Phandelver campaign without giving PCs too much XP?

What to do when moving next to a bird sanctuary with a loosely-domesticated cat?

A word that means fill it to the required quantity

Is bread bad for ducks?

Correct punctuation for showing a character's confusion

Why don't hard Brexiteers insist on a hard border to prevent illegal immigration after Brexit?

"as much details as you can remember"

Can you cast a spell on someone in the Ethereal Plane, if you are on the Material Plane and have the True Seeing spell active?

How to add class in ko template in magento2

Finding the area between two curves with Integrate

ELI5: Why they say that Israel would have been the fourth country to land a spacecraft on the Moon and why they call it low cost?

Will it cause any balance problems to have PCs level up and gain the benefits of a long rest mid-fight?

Is it ethical to upload a automatically generated paper to a non peer-reviewed site as part of a larger research?

Loose spokes after only a few rides

Why did Peik say, "I'm not an animal"?

Are there any other methods to apply to solving simultaneous equations?

Dropping list elements from nested list after evaluation

What was the last CPU that did not have the x87 floating-point unit built in?

How come people say “Would of”?

Why can't wing-mounted spoilers be used to steepen approaches?

What do these terms in Caesar's Gallic Wars mean?



How to handle columns with categorical data and many unique values



The 2019 Stack Overflow Developer Survey Results Are Indecision trees on mix of categorical and real value parametersPandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)Imputation of missing values and dealing with categorical valuesHow to deal with categorical variablesOne hot encoding error “sort.list(y)…”One hot encoding vs Word embeddingHow to implement feature selection for categorical variables (especially with many categories)?ML Models: How to handle categorical feature with over 1000 unique values“Binary Encoding” in “Decision Tree” / “Random Forest” AlgorithmsDealing with multiple distinct-value categorical variables










4












$begingroup$


I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.



I also have another column with 145 nunique values that I could also use in my model that represents product category.



Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?



Can you point me to the right direction if I should use another encoding also?










share|improve this question









$endgroup$
















    4












    $begingroup$


    I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.



    I also have another column with 145 nunique values that I could also use in my model that represents product category.



    Can I use one hot encoding to these columns or there's a problem with that solution?
    Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?



    Can you point me to the right direction if I should use another encoding also?










    share|improve this question









    $endgroup$














      4












      4








      4


      0



      $begingroup$


      I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.



      I also have another column with 145 nunique values that I could also use in my model that represents product category.



      Can I use one hot encoding to these columns or there's a problem with that solution?
      Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?



      Can you point me to the right direction if I should use another encoding also?










      share|improve this question









      $endgroup$




      I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.



      I also have another column with 145 nunique values that I could also use in my model that represents product category.



      Can I use one hot encoding to these columns or there's a problem with that solution?
      Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?



      Can you point me to the right direction if I should use another encoding also?







      machine-learning data categorical-data encoding






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Apr 8 at 11:04









      dungeondungeon

      394




      394




















          1 Answer
          1






          active

          oldest

          votes


















          5












          $begingroup$

          For categorical columns, you have two options :



          1. Entity Embeddings

          2. One Hot Vector

          For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.



          Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.



          Articles that explain Embeddings :



          • An Overview of Categorical Input Handling for Neural Networks


          • On learning embeddings for categorical data using Keras


          • Google Developers > Machine Learning > Embeddings: Categorical Input Data


          • Exploring Embeddings for Categorical Variables with Keras by Florian Teschner






          share|improve this answer











          $endgroup$













            Your Answer





            StackExchange.ifUsing("editor", function ()
            return StackExchange.using("mathjaxEditing", function ()
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            );
            );
            , "mathjax-editing");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "557"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            5












            $begingroup$

            For categorical columns, you have two options :



            1. Entity Embeddings

            2. One Hot Vector

            For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.



            Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.



            Articles that explain Embeddings :



            • An Overview of Categorical Input Handling for Neural Networks


            • On learning embeddings for categorical data using Keras


            • Google Developers > Machine Learning > Embeddings: Categorical Input Data


            • Exploring Embeddings for Categorical Variables with Keras by Florian Teschner






            share|improve this answer











            $endgroup$

















              5












              $begingroup$

              For categorical columns, you have two options :



              1. Entity Embeddings

              2. One Hot Vector

              For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.



              Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.



              Articles that explain Embeddings :



              • An Overview of Categorical Input Handling for Neural Networks


              • On learning embeddings for categorical data using Keras


              • Google Developers > Machine Learning > Embeddings: Categorical Input Data


              • Exploring Embeddings for Categorical Variables with Keras by Florian Teschner






              share|improve this answer











              $endgroup$















                5












                5








                5





                $begingroup$

                For categorical columns, you have two options :



                1. Entity Embeddings

                2. One Hot Vector

                For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.



                Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.



                Articles that explain Embeddings :



                • An Overview of Categorical Input Handling for Neural Networks


                • On learning embeddings for categorical data using Keras


                • Google Developers > Machine Learning > Embeddings: Categorical Input Data


                • Exploring Embeddings for Categorical Variables with Keras by Florian Teschner






                share|improve this answer











                $endgroup$



                For categorical columns, you have two options :



                1. Entity Embeddings

                2. One Hot Vector

                For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.



                Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.



                Articles that explain Embeddings :



                • An Overview of Categorical Input Handling for Neural Networks


                • On learning embeddings for categorical data using Keras


                • Google Developers > Machine Learning > Embeddings: Categorical Input Data


                • Exploring Embeddings for Categorical Variables with Keras by Florian Teschner







                share|improve this answer














                share|improve this answer



                share|improve this answer








                edited Apr 8 at 15:10

























                answered Apr 8 at 12:05









                Shamit VermaShamit Verma

                1,5291314




                1,5291314



























                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Data Science Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Sum ergo cogito? 1 nng

                    三茅街道4182Guuntc Dn precexpngmageondP